Lumina

For a single large service, in-product feedback forms collect thousands of comments a month, more than a team can read in full, and more arrive through support tickets, internal channels, and public forums such as AWS re:Post. Before Lumina, some design teams held recurring grooming sessions to read part of that feedback by hand over several weeks, trying to spot patterns and help product and engineering teams prioritize fixes; others had no time to read it at all. Themes and long-term trends were hard to see that way, so teams had only a partial picture of what customers were struggling with, and little to go on when deciding what to fix.

The second problem was structural. AWS offers more than 240 services, and teams work in service silos, each reading its own service's feedback. Many customer problems don't stay inside one service: a Lambda function timing out on its way to a serverless cache inside a VPC touches three teams' products at once. A theme like that has no single owner, and with no shared way to compare feedback across services, nobody was positioned to see it.

Lumina began in mid-2023 as a Jupyter notebook experiment using LangChain and Anthropic models on Amazon Bedrock. At the time, it was unclear whether large language models could read technical cloud feedback reliably. The notebook combined each message with its metadata into a short summary, generated categories, and compiled a report. It started as a small bet alongside my main work in the AWS Databases group. Once the prototype proved useful, it grew over about eight focused months of building across three versions: batch reports to test feasibility, comment-by-comment classification for specificity and trust, and ranked priorities with recommendations.

The first deployed version was a Step Functions workflow. A team uploaded a CSV of in-product feedback, and the workflow summarized the comments, sorted them into categories, and generated a report: an executive summary, overall issues and key insights, critical issues, emerging opportunities, and summaries per customer segment. Teams who had been reading a fraction of their feedback by hand now had a summary of every comment in the file in minutes.

Teams liked those early reports, but once they shared them, the conversation turned quickly to harder questions: where a claim came from, and whether it could be more specific. Knowing that customers struggled with configuration was too broad for an engineering team to act on; they needed to know which settings, on which engine, caused which failure. They also needed to see the evidence underneath the prose, including whether a claim rested on forty comments or two vocal ones, and what those customers actually wrote. Without a way to verify the claims, people hesitated to act on the summary.

Summarizing many comments in a single pass left the analysis to the model. It reported whatever caught its attention: the loudest themes, and comments near the start and end of a long input, while those buried in the middle quietly dropped out, a weakness of long-context models known as “lost in the middle.” Run twice on the same file, it could tell two different stories, and neither linked back to individual customer reports.

Giving the model one comment at a time, with a few narrow questions to answer, made the analysis far more repeatable: every comment was asked the same questions, got a precise summary, and kept a direct trail to the source. It also changed who did the sensemaking. Instead of the model deciding what mattered, its answers became data that could be counted, charted, and filtered, so teams could see the patterns for themselves and judge which ones deserved attention.

I rebuilt the pipeline to classify each comment as it arrives, along four independent lenses. I chose each for what it would let a team see:

• Product area: which features customers were talking about, the first thing every team wanted to know. It follows a multi-level feature map, from broad areas like monitoring or storage down to a specific workflow or engine feature.
• Architectural concern: the system concern underneath the comment, such as security, reliability, performance, or cost. Most of what customers do in the cloud comes back to one of a handful of these concerns.
• Customer intent: what the customer was trying to do, such as understanding a concept, solving a problem, making or validating a decision, or requesting a feature. It tells apart kinds of feedback that read alike on the surface.
• Adoption stage: where the customer was with the service, from getting started and implementation through migration and ongoing maintenance. It is the least precise lens, since comments rarely say so outright, but as data it still shows roughly what customers are doing: maintenance on mature services, and learning on newer ones.

Read together, the lenses trace the work behind a comment. A customer starts with a goal and breaks it into tasks. Each task happens at some stage of adopting a service, touches particular features, and depends on concerns like reliability or cost. When something along that path gets in the way, the customer writes. The comment is only the last step of that path, and the lenses try to capture the context around it.

For example, suppose a customer writes that a failover took longer than expected during a version upgrade. The pipeline stores that comment at the intersection of all four lenses: solving a problem, during maintenance, in high availability, under reliability. Because each lens answers a different question, different roles can enter the same record from their own angle, and someone new to a service can start with a broad product area and narrow down step by step to the individual comments.

The architectural-concern lens borrows from the AWS Well-Architected Framework, an established model of what every cloud system has to get right. Every AWS service names its features in its own dialect, so a permissions or failover issue looks different on the surface from one product to the next. Because the underlying concerns hold across services, a reliability failure in one service can be read alongside the same tension in another.

To build the product-area lens for a new service, I worked from its documentation and APIs to compile an inventory of its features and arrange them into the levels of the map. Onboarding a service this way takes roughly one to three days, and doing it across dozens of services is how I learned how cloud services are put together.

Onboarding services one at a time created a problem of its own. Each map grew from that service's documentation, and the feature names models proposed drifted from one service to the next: "Tagging" in one, "Resource Tagging" or "Tagging and Resource Organization" in another. Small differences like these quietly break any comparison across services. So I built a view of the whole taxonomy at once: every onboarded service as a column, every subcategory as a row, each cell counting that service's features there. Seen together, near-duplicates and names filed under two categories stood out, and I consolidated them into one shared vocabulary. Then I wrote those decisions down as a ruleset, so each new service's schema follows the same vocabulary from the start.

Classifying individual comments made them searchable, but teams still had to read across many records to decide what to fix first. The third version groups related comments using semantic embeddings alongside their classifications to identify recurring customer needs. It then turns those needs into evidence-backed priorities, ranked using comment metadata such as volume and severity, with suggested next steps for designers, engineers, product managers, or researchers. A priority represents the same underlying need, the same owner, and the same action, while broader themes connect priorities that share a surface but require different owners or actions.

To keep those priorities trustworthy, the pipeline splits the work between the language model and deterministic code. The model makes the semantic judgments: naming groups, clustering related needs, and drafting the rationale. Code then validates coverage, checks group assignments, and verifies every citation, so every quote in a priority stays linked to its verbatim comment. Because language models can group the same items differently from run to run, I tested the pipeline on repeated runs over the same feedback. The exact groupings shifted, but the top issues and the first recommended action generally held. Making the synthesis fully repeatable is still open, and I’m not sure it is the right goal: a synthesis that is reliably close already gives teams something they can act on.

Because every service is classified against the same architectural concerns, per-service priorities also roll up into a cross-service view, where a theme that runs through many services appears as one pattern instead of scattered across separate teams' reports.

Alongside ranked priorities, this version added semantic search, so someone can start from one comment and find similar problems across services and channels, trend lines per classification, and a broader trends view that summarizes which friction points are rising and which ones recent releases have eased.

I designed and built the system end to end on serverless AWS: Lambda functions orchestrated by Step Functions, Anthropic models on Amazon Bedrock, and DynamoDB for the records. A few architectural decisions carried it from a report generator for one team to a system serving more than 50 services:

• Every feedback item becomes a standard record. Each source has a small adapter that turns its export, a CSV of in-product feedback or JSON from forum posts, into a common record that keeps the comment alongside its metadata: service, source, date, and customer segment. Everything downstream reads only that format, so connecting a new channel as teams onboarded left classification and the interface untouched.
• Events and schedules drive the stages. Nobody runs the pipeline by hand. S3 events and EventBridge start each stage as new data lands or records change, and scheduled jobs generate reports for different time frames.
• Classify once, query many times. Within classification, a Step Functions Map state fans out over comments, with each lens as its own model call and its own prompt, so one lens can be changed and re-evaluated without touching the others. Each result is written back onto its record in DynamoDB, with global secondary indexes for the interface’s queries. Because filtering, cross-tabulating, and ranking by volume and severity read those stored attributes directly, these views never invoke a model at query time: they stay fast and cheap, and the same filters over the same data return the same results.
• Service taxonomies live as data. Each service’s feature map is a schema the classifiers load as context at runtime, written to the shared vocabulary’s rules, so onboarding a service means adding its schema while the pipeline stays as it is.
• Later features are independent readers. Priorities, trend tracking, and semantic search each read the same classified records on their own, so I could add them later without changing how comments are ingested or classified.

AI helped me write the pipeline faster, and as the tools improved, they took on larger pieces of the work. Still, to judge what the AI produced and keep up with its changing limits, I had to deepen my understanding of distributed systems.

Much of the job happened outside the code. I worked with partner teams, got permission from each data owner to read their sources, and took Lumina through security review. Because non-deterministic models do not give the same answer every time, I also had to verify how the system read feedback. I studied emerging ways to handle synthesis with these models, built evaluations, and tuned the system over several rounds so the results were accurate, reliable, and trustworthy. To decide what to build in each version, I talked with the first people using Lumina about what they found useful.

Feedback serves two needs: understanding what customers are struggling with, and knowing which areas need fixing and what exactly to fix. Lumina supports both from the same classified records, built from every comment rather than a sample, and each role reads them for its own questions:

• Designers find the parts of a product where frustration concentrates, the specific issues within them, and scenarios a design missed.
• Researchers find gaps worth studying.
• Product managers draw on customer quotes and evidence of pain while writing PR/FAQs, Amazon’s working-backwards proposals, for new initiatives.
• Engineers spot bugs in newly shipped functionality as customers report them.
• Everyone moves faster on everyday tasks, from pulling background context before a project to finding real customer examples for a design review. Monthly feedback analysis that took a team several weeks now takes a few hours.

Lumina now covers more than 50 AWS services, and use has spread from the original design team to engineering and product partners, with more teams onboarding their own data sources. I have also shared it with the wider design community at Amazon, in several design talks and internal conferences.

The way people work is changing faster than any tool can settle, so I built Lumina as a foundation: classified, cited feedback that can feed whichever tools and workflows teams use, and stay useful whatever those tools turn out to be. The work now is bringing it into the workflows where teams plan, design, and fix, and connecting it with the other systems they rely on, so customer evidence is at hand wherever a decision is being made, and people stay the ones making it.