Ai Operations Analyst

At Johnson prioritise findings and escalate complex defects to engineering teams.
• Own incident analysis and follow-up for assigned operational issues, contribute to root-cause reviews, and implement agreed preventive actions within the role's technical authority.
LLMOps release approval remains with accountable owners.
• Maintain consistent use of shared operational tools and identify opportunities to standardise support processes across models, frameworks, and agent patterns.
Support engineers and product teams during design by helping define telemetry, alerting, troubleshooting, and recovery requirements; design decisions remain with accountable engineering and product owners.
• Analyse functional evaluation results supplied by the Agent Factory or AI product teams alongside operational signals, flag regressions, and support readiness reviews; functional quality ownership remains with those teams.
• Own the accuracy and maintenance of assigned runbooks, checklists, knowledge articles, and reusable operational examples.
AI Consumption budget and investment decisions remain with accountable leads.
Security permission changes and secrets administration remain with authorised owners.
• Triage and investigate runtime security alerts, including suspected prompt injection, unauthorised tool use, data leakage, and anomalous access; coordinate response with security and engineering teams.
• Validate that operational logging, retention, redaction, and access practices follow approved requirements, and resolve or escalate identified deviations.
• Maintain complete and accurate operational evidence linking versions, configurations, identities, actions, approvals, and outcomes.
• Coordinate recurring policy checks and evidence collection, analyse gaps, and track remediation to closure with accountable control owners.
• Contribute to security testing and incident response through operational investigation, evidence analysis, and delivery of assigned remediation actions.
Operational Service Support contribute to design reviews by highlighting operational lessons, support needs, and potential service risks.
• Promote adoption of approved operational practices and resolve routine process questions, escalating architectural or policy decisions to accountable owners.
Governance, Risk track the outcomes of agreed actions.
• Maintain an accurate inventory of production AI services and assess coverage gaps in telemetry, alerts, runbooks, cost controls, and security guardrails.
Qualifications Required • Bachelor's degree in Computer Science, Engineering, Data Analytics, Artificial Intelligence, or a related discipline, or equivalent relevant experience.
• 3-5 years of relevant experience in IT operations, application support, operational analytics, DevOps, or a related technical area, including hands-on responsibility for monitoring, incident investigation, or service reporting.
• Practical understanding of AI or machine learning applications, with experience supporting AI-enabled services or analysing their operational data.
• Working proficiency with scripting language, APIs, and structured data analysis, including automation of routine operational tasks.
• Hands-on familiarity with cloud services, version control, and software delivery processes; ability to work within established testing and change-management practices.
• Experience using dashboards, logs, metrics, and traces to investigate incidents, explain service trends, and improve operational reporting.
• Working understanding of generative AI application patterns, including prompts, model responses, RAG, and tool calling, and their implications for monitoring and troubleshooting.
• Working knowledge of secure data handling, access controls, privacy-aware logging, and operational audit evidence.
• Ability to independently manage assigned operational activities, investigate ambiguous issues, recommend improvements, and recognise when specialist escalation is required.
• Clear technical communication, effective cross-team coordination, and strong documentation and follow-through skills.
Preferred • Experience in regulated environments with security, privacy, quality, risk, or audit requirements.
• Experience supporting an AI application, RAG workflow, chatbot, or agent framework in a live service environment.
• Experience with consumption reporting, cloud cost analysis, dashboards, and tracking the impact of optimisation actions.
• Experience with incident and problem management, service-level objectives, deployment-readiness checks, CI/CD, or infrastructure-as-code workflows.
Required Skills: Preferred Skills: AI Judgement Integration, Analytical Reasoning, Coaching, Communication, Cross-Functional Collaboration, Demand Forecasting, Human-Computer Interaction (HCI), Persistence and Tenacity, Product Costing, Product Development, Product S