Engineer - Web Data Collection

The Web Data Collection Engineer is the backbone of the Web Data Acquisition department at Engage3. In this role, you are responsible for designing, building, and maintaining robust web scrapers that provide critical data to our customers and the data that allows our data science and reporting teams to provide core and crucial insights to our retail clients.

The Tech

At Engage3 we use our own Mozenda platform for the great majority of our web scraping work rather than open source frameworks like Playwrite or Selenium. We do this because it is robust, fast, and allows for relatively easy implementation of simple web scraping agents while still maintaining the flexibility to use advanced techniques to optimize for high-volume collection and avoiding and bypassing CAPTCHA services.

 

It also provides a degree of consistency and infrastructure that has allowed us to build and maintain thousands of web scrapers over time with a relatively small team.

 

The data we collect for large retail clients feeds into our Snowflake database. Familiarity with and/or willingness and ability to learn SQL well enough to query collected data and troubleshoot problems in the collection process is highly encouraged and expected of more senior team members.

 

General Expectations

• B2 or above English proficiency
• Be available in-office, Monday-Friday, during general business hours (generally 08:00 - 16:30 or 09:00 - 17:30)
• Respect and follow the directions of your team leader
• Respect and develop good working relationships with other team members and with colleagues in other departments
• A strong ability and willingness to learn and improve in a collaborative and self-directed way

 

Skill and Education Requirements

• Trainee Level
  • Basic familiarity with HTML, CSS, and JavaScript
  • A drive to learn new things and work creatively to solve problems
  • Strong attention to detail
  • Ability to work on challenging problems without giving up
  • Bachelor’s degree in CS, IT, Computer Engineering or current enrollment in a technical degree program (preferred, not required)

 

• Associate Level (after formal training and after the initial trial period)
  • Ability to use Mozenda to build most web scraping projects
  • Ability to write XPath and straightforward RegEx in Mozenda agents
  • Strong knowledge of the Engage3 Retail data specification
  • Ability to identify straightforward cases where a site uses APIs to load information and the ability to replicate and parse those requests using JavaScript