I'm not sure you need queues to do this

I would set a "last_scraped_at" timestamp against each url you scrape

Then in your cron:

• select ~100 urls and order by last_scraped_at ASC
• scrape and process each url

This way you will always process the oldest url first - you could add a priority field and sort by this first to allow you to always process certain urls

On cron jobs like this where you do not know how long it might take to process I have found lock files to pretty useful to prevent multiple instances running at the same time. Just check for the lock file at when the job starts, if it is locked quit out otherwise, lock the file and run your script (remember to unlock the file at the end :))

http://php.net/manual/en/function.flock.php