ML Pipeline Assessment & Optimization Project

We're seeking an ML Engineer to conduct a comprehensive assessment and optimization of TrackGBV—an AI-powered system that analyzes court sentencing decisions in gender-based violence (GBV) cases across 12 Pacific Island nations. This project uses Claude API, Python, CockroachDB, AWS S3, and Fly.io to extract structured data from court transcripts, exposing patterns of judicial bias to inform policy reform.

This is a defined assessment and improvement project with the following deliverables:

Phase 1: System Assessment (Weeks 1-3)
- Conduct a technical audit of the AI extraction pipeline to identify why certain legal features have lower accuracy
- Analyze QC feedback patterns to pinpoint systematic extraction errors
- Deliver: Assessment report with prioritized recommendations

Phase 2: Prompt Optimization (Weeks 4-9)
- Refine AI prompts for the highest-priority legal data fields based on legal team feedback
- Implement improvements to address systematic extraction errors identified in QC review
- Test and validate prompt changes against sample cases
- Deliver: Optimized prompt library with documented changes and performance benchmarks

Phase 3: Evaluation Infrastructure (Weeks 10-12)
- Design metrics and workflows for ongoing accuracy measurement
- Create feedback loop between QC review and prompt refinement
- Deliver: Evaluation framework and roadmap for future improvements

Skills needed:
- Experience with LLM prompt engineering and evaluation
- Strong Python skills
- Familiarity with cloud infrastructure (AWS S3, databases)
- Ability to translate domain expert feedback into technical solutions
- Clear technical writing skills

Note: This work involves reviewing content related to violence against women and requires a trauma-informed approach. Given the sensitive nature of court data, this role requires strong attention to data security, access controls, and ethical handling of legal records.

ICAAD (International Center for Advocates Against Discrimination) is a human rights advocacy center that equips those most harmed by systemic inequity with the tools to drive systemic change. We use our expertise to help communities and governments create a more equitable future. Though we're a small organization, we achieve outsized impact through partnerships with committed pro bono professionals, including past Taproot volunteers with whom we maintain strong ongoing relationships.

TrackGBV is our flagship initiative to expose judicial bias in GBV cases across 12 Pacific Island nations. It's the first systematic effort to reveal how courts handle violence against women, documenting when judges minimize harm, accept discriminatory "mitigating factors," or apply gender stereotypes in sentencing. Our TrackGBV data dashboard (https://icaad.ngo/trackgbv%20dashboard/) makes these patterns visible and accessible, and our evidence has already contributed to policy changes in 2 countries.

Your impact: This assessment and optimization project is critical to scaling TrackGBV beyond the Pacific region. By improving extraction accuracy and creating robust documentation, you'll enable us to process more cases reliably, expand to new countries efficiently, and provide credible data-driven evidence for policy reform. Our findings inform judicial training programs, advocacy briefs, and legal reforms.

Building long-term capacity: This project will strengthen ICAAD's technical foundation, making it easier to onboard future volunteers, maintain the system independently, and attract ongoing technical support. For volunteers who find the mission compelling, successful completion of this project often leads to continued advisory relationships—several past Taproot volunteers have become long-term ICAAD collaborators.

This isn't just an assessment. You'll ensure that patterns of discrimination hidden in court transcripts become visible, actionable data that drives real change for women across the Pacific and beyond.

We've laid strong groundwork for your success:

Complete system handover: We have comprehensive technical documentation covering architecture, deployment processes, and system functionality, and our departing lead developer will be available to answer questions during the project
Clear objectives: We have defined priorities for system improvements and a backlog of QC feedback ready for prompt refinement work
Collaborative structure: You'll be working alongside our in-house and pro bono legal team who conduct quality control and provide domain expertise on the complex legal features being extracted. Understanding these features is essential for translating their feedback into prompt refinements.
Existing infrastructure: The pipeline is fully operational—you're assessing and optimizing, not building from scratch
Direct access: You'll work directly with me (Director of Data Analytics & Insights) for strategic direction and our legal team for domain expertise
Resources ready: We have established workflows, cost tracking mechanisms, and quality control interfaces already in place

Success in this project looks like: A comprehensive assessment report, optimized prompts with documented performance improvements, updated technical documentation, and a clear roadmap that positions ICAAD to maintain and scale the system independently or with future volunteer support.

What you can expect from us:
- Prompt responses to questions and regular check-ins
- Access to all documentation, codebase, and system credentials
- Clear communication about priorities and organizational needs
- Flexibility in how you structure your time within the project timeline
- Genuine appreciation for your expertise and partnership