5 days old

AI/ML - Engineering Program Manager, Search Human Annotation

Seattle, WA 98104
  • Job Code
    200223703
Summary

Summary

Posted: Apr 5, 2021

Weekly Hours: 40

Role Number:200223703

Imagine what you could do here. At Apple, new ideas have a way of becoming extraordinary products, services, and...Summary

Summary

Posted: Apr 5, 2021

Weekly Hours: 40

Role Number:200223703

Imagine what you could do here. At Apple, new ideas have a way of becoming extraordinary products, services, and customer experiences very quickly. Do you love taking on challenges that create a positive impact? Are you passionate about empowering many ground-breaking intelligent experiences to be made? We are looking for people like you!

The Siri Search team mission is to deliver compelling information seeking experiences for Apple users. The team continuously pushes for innovating core technologies to empower a great user experience across multi search modalities. The team powers search products on Safari, Spotlight, Siri Assistant, News App, Lookup and many more. A critical enabler for continuous Search products improvement is scalable, rigorous and comprehensive search quality evaluation. We're looking for someone to build and lead the program for human annotation to empower deep and precise quality evaluation that supports multi search modality, both at component level and end product level, across on-device and server search.

Key Qualifications

  • 5+ years experience managing a large program that consists of cross functional product, data science or software engineering teams
  • Proven capability to set direction and work across a large number of partners to implement process and efficient outcomes in a data-driven environment
  • Experience in experimental design, industry research, or academic research
  • Self-motivated and proactive with demonstrated process optimization capabilities required to forge a path to success
  • Phenomenal attention to detail and organized
  • Strong curiosity and creativity in solving challenges surfaced in human annotations across different human evaluation scenarios
  • Extraordinary communication and presentation skills, written and verbal, to all levels of an organization
  • The ability of building a new program/process from the group up including defining requirements, driving agreements from multiple teams and executing towards a successful completion

Description

In this role you will lead Search human annotation program roadmap planning, annotation strategy, budget, partnership management to scale Siri human annotation to support the next generation of search products via human annotation based evaluation excellence. Additionally, you will be the specialist on human annotation, raising the bar for effective and efficient utilization of human judgment in empowering better product evaluation and development in the AI/ML org. You will work with world-class data scientists, machine learning engineers, evaluation tooling development engineers, Human Annotation Operation team and product teams to build the highest quality user experience that over 1 billion Apple customers love.
- Lead roadmap, detailed execution of future human annotation platforms and programs to deliver world class search quality offline evaluation via close collaboration with data scientists in Search analytics team
- Contribute to brainstorming on continuous human evaluation methodology improvement as well as solutions and workarounds for human evaluation challenges (e.g. on-device search evaluation)
- Drive organization wide initiatives to scale human annotation tooling and process
- Lead allocations of grading resources across Siri teams
- Partner with procurement and vendors to meet human annotation resource demand
- Collaborate with Search analytics team, Engineering team, Privacy team, Siri Annotation Operation team, Evaluation Tooling and Platform teams to manage dependencies, requirements as well as executions to ensure excellence in human grading based quality evaluation
- Organize data sharing initiatives for Siri teams to maximize utilization and value generation of our human grading data assets
- Establish efficacy and efficiency standard methodologies for human annotation in Siri org

Education & Experience

BA/BS, MA/MS or PhD in Linguistics, Communication Studies, Journalism, Information Science, Computer Science, Computer Engineering or related fields/experiences

Additional Requirements

  • 3+ years experience in leading large scale human annotation efforts or working on ML projects that involve large scale of human annotation data collection is a strong plus


Posted: 2021-04-06 Expires: 2021-05-06

Before you go...

Our free job seeker tools include alerts for new jobs, saving your favorites, optimized job matching, and more! Just enter your email below.

Share this job:

AI/ML - Engineering Program Manager, Search Human Annotation

Apple, Inc.
Seattle, WA 98104

Join us to start saving your Favorite Jobs!

Sign In Create Account
Powered ByCareerCast