Job Description

Our client, a leading data engineering company is looking for Japanese speaking LLM Evaluation Specialists for a short-term contract position with an immediate start.

• Variable hours, Must be available Monday to Friday between 8am-6pm for up to 8 hour shifts (note hours of work may vary per day).
• Pay rate: $18.23/hour
• Start date: Monday, June 16
• Contract: 10 days (Possibility of extension, based on project needs)

Please note: You must have your own laptop with a secure internet connection. Additionally, if selected you will be required to complete a background and reference check.

100% REMOTE ANYWHERE IN CANADA - This is an assignment that can be worked from the comfort of your home.

We are looking for linguistically and culturally aware professionals to support the evaluation and enhancement of multilingual prompt-response datasets for large language models (LLMs). This role involves rubric design, evaluation of translations and model outputs, prompt creation, and red teaming focused on identifying and surfacing cultural nuances and biases in LLM behavior.

Key Responsibilities:

• Update rubric definitions with region/language-specific examples to ensure cultural and linguistic relevance.
• Identify the need for additional rubrics tailored to specific languages or regional contexts.
• Review prompts translated from English into the target language and revise where translations appear unnatural or inaccurate.
• Writing of thoughtful prompts which can test the cultural awareness of LLM models.
• Rate prompt-response pairs using a standardized evaluation template based on rubrics and provide detailed justifications to base the findings.
• Document problematic outputs and annotate them with clear explanations of rubric violations or cultural insensitivities.

Required Qualifications:

• Native proficiency in Japanese and deep familiarity with cultural norms in the corresponding region.
• Experience in LLM evaluation, content moderation, linguistic QA, writing, teaching, marketing, blogging, or journalism required.
• Bachelor's degree required.
• Strong attention to detail with the ability to identify subtle issues in language use, tone, and cultural references.
• Comfortable working in spreadsheets and evaluation templates.
• Ability to work independently and meet deadlines.
• Attention to detail and commitment to delivering high-quality work.
• Flexibility to adapt to changing project requirements and priorities.
• Collaborative mindset and willingness to work with a multidisciplinary team.
• Own a laptop with a secure internet connection.

Please apply below if you meet the qualifications!