Built at Scale Notes on systems that work
← All projects
Building aws-log-detective-ai

Log Analytics Agent

Ask a plain-English question — "why did my EMR job on cluster j-ABC123 fail?" — and a team of specialist agents investigates across your AWS analytics engines, then returns a root-cause answer with a recommended fix.

View on GitHub Related notes

How it works.

The system treats an investigation like an on-call engineer would: form a plan, gather evidence from whichever engines are relevant, then reason over it — grounded against a library of known failure patterns.

Log Analytics Agent architecture A plain-English question flows into an orchestrator, which dispatches specialist agents across six AWS analytics engines; a synthesizer grounded by a Chroma vector store consolidates the evidence into a root-cause answer. Plain-English question “why did my EMR job on j-ABC123 fail?” Orchestrator plans engines · parallel + sequential specialist agents · read-only EMR CloudWatch Logs Athena OpenSearch Redshift S3 log files Synthesizer merges evidence · consolidates Chroma store 160 failure patterns Root cause + recommended fix with supporting evidence and gaps

Planning is structured through forced tool calls; each specialist fails independently so one dead engine never sinks the whole investigation.

What it does.

Built to turn “grep through terabytes of logs” into a conversation.

Multi-agent orchestration

A lead orchestrator plans the investigation, specialist agents query each engine, and a synthesizer consolidates the findings.

Parallel + sequential

Independent queries are dispatched concurrently via a thread pool; dependent steps (e.g. reading the actual S3 log file) wait on upstream evidence.

Six analytics engines

CloudWatch Logs Insights, Athena, EMR, S3 log files, OpenSearch, and Redshift — pick the right tool per question.

RAG-grounded synthesis

Answers are grounded against a Chroma vector store of 160 public failure patterns for Spark, EMR, S3, and CloudWatch.

Pluggable LLM backend

Runs on the Anthropic API or Amazon Bedrock, switched with a single environment variable.

Read-only + safe

Every tool operates in strict read-only mode. Context is optimized with capped S3 reads (~200 KB) to stay inside the window.

Built with.

Python 3.10+ with the Anthropic SDK, over the AWS analytics stack.

AWS services
CloudWatch Logs InsightsAmazon AthenaAmazon EMRAmazon S3OpenSearchAmazon RedshiftAmazon Bedrock
Key dependencies
anthropic[bedrock]boto3chromadbrich

Try it.

A CLI and a Python API, both read-only.

# install
git clone https://github.com/mannem/aws-log-detective-ai.git
cd aws-log-detective-ai
pip install -e .

# ask a question
log-detective "why did my EMR job on cluster j-ABC123 fail?"

# scope engines + get JSON, on Bedrock
log-detective "find 5xx spikes in my ALB logs in the last 6h" \
    --engines athena,cloudwatch --backend bedrock --json
from log_detective import Orchestrator, Settings

orch = Orchestrator(Settings.from_env())
result = orch.investigate("why did my emr job j-ABC123 fail?")
print(result.answer)
View on GitHub