Build an internal collector
An internal collector gives the engineering team control over inputs, extraction logic, validation, storage, and delivery.
It may be suitable when:
- The scope is small
- The work is experimental
- The team has relevant engineering experience
- The required fields are limited
- The team accepts ongoing maintenance
The first working script is not the complete system. A production collector also needs:
- Logging
- Retry rules
- Failed-record reporting
- Schema tests
- Duplicate handling
- Collection timestamps
- Monitoring for page changes
- A process for reviewing incomplete records
Without these controls, a job may continue running while returning partial or inconsistent data.



