AI Evaluation & Assurance Architect

Job Description

Purpose of the role

To drive technical excellence and innovation by leading the design and implementation of robust software solutions, providing mentorship to engineering teams, fostering cross-functional collaboration, and contributing to strategic planning to ensure the delivery of high-quality solutions aligned with business objectives.

Accountabilities

• Provision of guidance and expertise to engineering teams to ensure alignment with best practices and foster a culture of technical excellence.
• Contribution to strategic planning by aligning technical decisions with business goals, anticipating future technology trends, and providing insights to optimize product roadmaps.
• Design and implementation of complex, scalable, and maintainable software solutions, considering long-term viability and business objectives.
• Mentoring and coaching to junior and mid-level engineers to foster professional growth and knowledge sharing, elevating the overall skillset and capabilities of the organization.
• Collaboration with business partners, product managers, designers, and other stakeholders to translate business requirements into technical solutions and ensure a cohesive approach to product development.
• Innovation within the organization by identifying and incorporating new technologies, methodologies, and industry practices into the engineering process.

Vice President Expectations

• To contribute or set strategy, drive requirements and make recommendations for change. Plan resources, budgets, and policies; manage and maintain policies/ processes; deliver continuous improvements and escalate breaches of policies/procedures..
• If managing a team, they define jobs and responsibilities, planning for the department’s future needs and operations, counselling employees on performance and contributing to employee pay decisions/changes. They may also lead a number of specialists to influence the operations of a department, in alignment with strategic as well as tactical priorities, while balancing short and long term goals and ensuring that budgets and schedules meet corporate requirements..
• If the position has leadership responsibilities, People Leaders are expected to demonstrate a clear set of leadership behaviours to create an environment for colleagues to thrive and deliver to a consistently excellent standard. The four LEAD behaviours are: L – Listen and be authentic, E – Energise and inspire, A – Align across the enterprise, D – Develop others..
• OR for an individual contributor, they will be a subject matter expert within own discipline and will guide technical direction. They will lead collaborative, multi-year assignments and guide team members through structured assignments, identify the need for the inclusion of other areas of specialisation to complete assignments. They will train, guide and coach less experienced specialists and provide information affecting long term profits, organisational risks and strategic decisions..
• Advise key stakeholders, including functional leadership teams and senior management on functional and cross functional areas of impact and alignment.
• Manage and mitigate risks through assessment, in support of the control and governance agenda.
• Demonstrate leadership and accountability for managing risk and strengthening controls in relation to the work your team does.
• Demonstrate comprehensive understanding of the organisation functions to contribute to achieving the goals of the business.
• Collaborate with other areas of work, for business aligned support areas to keep up to speed with business activity and the business strategies.
• Create solutions based on sophisticated analytical thought comparing and selecting complex alternatives. In-depth analysis with interpretative thinking will be required to define problems and develop innovative solutions.
• Adopt and include the outcomes of extensive research in problem solving processes.
• Seek out, build and maintain trusting relationships and partnerships with internal and external stakeholders in order to accomplish key business objectives, using influencing and negotiating skills to achieve outcomes.

All colleagues will be expected to demonstrate the Barclays Values of Respect, Integrity, Service, Excellence and Stewardship – our moral compass, helping us do what we believe is right. They will also be expected to demonstrate the Barclays Mindset – to Empower, Challenge and Drive – the operating manual for how we behave.

Join us at Barclays as an AI Evaluation & Assurance Architect -We’re looking for a forward-thinking architect to shape how AI is evaluated, governed, and trusted across the enterprise.

This is a foundational position within our AI Accelerator, focused on building and scaling frameworks that ensure AI systems are reliable, trustworthy, and defensible. You’ll define evaluation standards, assurance processes, and reference architectures across Generative AI, Agentic AI, and AI/ML—supporting their full lifecycle from design through to live operation. Working across architecture, engineering, risk, and governance, you’ll enable rapid AI adoption while maintaining control, transparency, and confidence.

To be successful as an AI Evaluation & Assurance Architect, you should have experience with

• Evaluation architecture for probabilistic AI: Design robust evaluation strategies for non-deterministic systems (e.g. LLMs), including test datasets, scoring methods, and release gates.

• Risk-based assurance design: Translate AI risks into effective controls, covering areas like bias, hallucination, adversarial testing, and data leakage.

• Operationalisation (LLMOps & monitoring): Embed evaluation into production through continuous monitoring, observability, CI/CD gating, and auditability.

Some other highly valued skills may include

• Simulation & synthetic data: Create realistic scenarios and stress tests to improve coverage and resilience.

• Statistics & measurement: Apply sampling, confidence metrics, and evaluation consistency techniques.

• Governance literacy: Produce assurance evidence, align with model risk frameworks, and support independent validation practices.

You may be assessed on the key critical skills relevant for success in role, such as risk and controls, change and transformation, business acumen strategic thinking and digital and technology, as well as job-specific technical skills

Location

This role is based in Knutsford / Glasgow / Manchester