The main objective of this project was to automate new URLs gathering and sorting. Number of PERL "crawler" scripts was written to achieve this goal.

W3Control Internet Database consists of hundreds of thousands of URLs that have been placed into 59 categories and is included with W3Control products W3Thermo and W3Filter. W3Thermo uses the database to categorize accessed sites when analyzing firewall and proxy logs. W3Filter software uses the database to allow or deny access to sites depending on the rules defined. W3Thermo and W3Filter can automatically receive updates to the database across the Internet each day.