AWS Machine Learning

OCTOBER 29, 2024

Observability refers to the ability to understand the internal state and behavior of a system by analyzing its outputs, logs, and metrics. Evaluation, on the other hand, involves assessing the quality and relevance of the generated outputs, enabling continual improvement.

Best practices 113

Best practices Feedback Metrics APIs 113

AWS Machine Learning

FEBRUARY 12, 2025

This approach allows organizations to assess their AI models effectiveness using pre-defined metrics, making sure that the technology aligns with their specific needs and objectives. The introduction of an LLM-as-a-judge framework represents a significant step forward in simplifying and streamlining the model evaluation process.

Metrics 104

Metrics Engineering Benchmark APIs 104

AWS Machine Learning

MARCH 11, 2025

How do Amazon Nova Micro and Amazon Nova Lite perform against GPT-4o mini in these same metrics? The following table provides example questions with their domain and question type. Amazon Bedrock APIs make it straightforward to use Amazon Titan Text Embeddings V2 for embedding data. get("message", {}).get("content")

Benchmark 111

Benchmark APIs Enterprise Scripts 111

AWS Machine Learning

MARCH 6, 2025

Current RAG pipelines frequently employ similarity-based metrics such as ROUGE , BLEU , and BERTScore to assess the quality of the generated responses, which is essential for refining and enhancing the models capabilities. More sophisticated metrics are needed to evaluate factual alignment and accuracy.

Metrics 118

Metrics Enterprise APIs Engineering 118

AWS Machine Learning

DECEMBER 2, 2024

Use faster auto scaling metrics – Take advantage of more granular auto scaling metrics like ConcurrentRequestsPerCopy to more accurately monitor and react to changes in inference traffic. It’s a dynamic policy that adjusts the number of copies based on a specified metric, such as CPU utilization or request count.

APIs 108

APIs Best practices Metrics Engineering 108

AWS Machine Learning

MARCH 25, 2025

Then we deep dive into the new rolling update feature for inference components and provide practical examples using DeepSeek distilled models to demonstrate this feature. Consider an example where a customer has 10 copies of an inference component spread across 5 ml.p4d.24xlarge You can find the example notebook in the GitHub repo.

APIs 110

APIs Engineering Accountability Metrics 110

AWS Machine Learning

AUGUST 2, 2024

adds new APIs to customize GraphStorm pipelines: you now only need 12 lines of code to implement a custom node classification training loop. To help you get started with the new API, we have published two Jupyter notebook examples: one for node classification, and one for a link prediction task. Specifically, GraphStorm 0.3

APIs 117

APIs Benchmark Construction Enterprise 117

AWS Machine Learning

DECEMBER 9, 2024

However, keeping track of numerous experiments, their parameters, metrics, and results can be difficult, especially when working on complex projects simultaneously. For example, you can give users access permission to download popular packages and customize the development environment. config_yaml = f""" SchemaVersion: '1.0'

Metrics 104

Metrics APIs Engineering Accountability 104

AWS Machine Learning

APRIL 10, 2025

Fine-tune an Amazon Nova model using the Amazon Bedrock API In this section, we provide detailed walkthroughs on fine-tuning and hosting customized Amazon Nova models using Amazon Bedrock. To do so, we create a knowledge base. The following diagram illustrates the solution architecture. For Job name , enter a name for the fine-tuning job.

APIs 124

APIs Metrics Best practices Construction 124

AWS Machine Learning

MARCH 27, 2025

We also showcase a real-world example for predicting the root cause category for support cases. For the use case of labeling the support root cause categories, its often harder to source examples for categories such as Software Defect, Feature Request, and Documentation Improvement for labeling than it is for Customer Education.

Education 111

Education Engineering APIs Enterprise 111

AWS Machine Learning

FEBRUARY 21, 2025

This serves as an example of how generative AI can streamline operations that involve diverse data types and formats. The solution uses the FMs tool use capabilities, accessed through the Amazon Bedrock Converse API. Use case and dataset For our example use case, we examine a patient intake process at a healthcare institution.

APIs 93

APIs Healthcare Consulting Consulting 93

AWS Machine Learning

APRIL 10, 2025

For instance, Pixtral Large is highly effective at spotting irregularities or insightful trends within training loss curves or performance metrics, enhancing the accuracy of data-driven decision-making. By choosing View API , you can also access the model using code examples in the AWS Command Line Interface (AWS CLI) and AWS SDKs.

APIs 109

APIs Management Marketing Engineering 109

AWS Machine Learning

JANUARY 28, 2025

To effectively optimize AI applications for responsiveness, we need to understand the key metrics that define latency and how they impact user experience. These metrics differ between streaming and nonstreaming modes and understanding them is crucial for building responsive AI applications.

Benchmark 77

Benchmark APIs Engineering Metrics 77

AWS Machine Learning

SEPTEMBER 28, 2023

As an example, time-series forecasting allows retailers to predict future sales demand and plan for inventory levels, logistics, and marketing campaigns. In this post, we describe the enhancements to the forecasting capabilities of SageMaker Canvas and guide you on using its user interface (UI) and AutoML APIs for time-series forecasting.

APIs 119

APIs Construction Finance Enterprise 119

AWS Machine Learning

NOVEMBER 20, 2024

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI.

Construction 117

Construction APIs Metrics Accountability 117

AWS Machine Learning

FEBRUARY 18, 2025

During these live events, F1 IT engineers must triage critical issues across its services, such as network degradation to one of its APIs. This impacts downstream services that consume data from the API, including products such as F1 TV, which offer live and on-demand coverage of every race as well as real-time telemetry.

APIs 70

APIs Engineering Metrics Big data 70

AWS Machine Learning

MARCH 3, 2025

Designed for both image and document comprehension, Pixtral demonstrates advanced capabilities in vision-related tasks, including chart and figure interpretation, document question answering, multimodal reasoning, and instruction followingseveral of which are illustrated with examples later in this post. Lets explore an example.

Benchmark 109

Benchmark APIs Enterprise Construction 109

AWS Machine Learning

NOVEMBER 13, 2024

For example, searching for a specific red leather handbag with a gold chain using text alone can be cumbersome and imprecise, often yielding results that don’t directly match the user’s intent. Amazon Titan FMs provide customers with a breadth of high-performing image, multimodal, and text model choices, through a fully managed API.

Engineering 115

Engineering Management APIs Healthcare 115

AWS Machine Learning

SEPTEMBER 20, 2024

Amazon Bedrock agents use LLMs to break down tasks, interact dynamically with users, run actions through API calls, and augment knowledge using Amazon Bedrock Knowledge Bases. In this post, we demonstrate how to use Amazon Bedrock Agents with a web search API to integrate dynamic web content in your generative AI application.

APIs 106

APIs Chatbots Construction Engineering 106

AWS Machine Learning

NOVEMBER 14, 2024

The user’s request is sent to AWS API Gateway , which triggers a Lambda function to interact with Amazon Bedrock using Anthropic’s Claude Instant V1 FM to process the user’s request and generate a natural language response of the place location. Here is an example from LangChain.

APIs 111

APIs Engineering Personalization Transportation 111

AWS Machine Learning

MARCH 14, 2025

Although automated metrics are fast and cost-effective, they can only evaluate the correctness of an AI response, without capturing other evaluation dimensions or providing explanations of why an answer is problematic. Human evaluation, although thorough, is time-consuming and expensive at scale.

Metrics 104

Metrics Best practices Engineering APIs 104

AWS Machine Learning

FEBRUARY 1, 2023

Amazon Rekognition has two sets of APIs that help you moderate images or videos to keep digital communities safe and engaged. Some customers have asked if they could use this approach to moderate videos by sampling image frames and sending them to the Amazon Rekognition image moderation API.

APIs 95

APIs Scripts Metrics Surveys 95

AWS Machine Learning

SEPTEMBER 7, 2023

In February 2022, Amazon Web Services added support for NVIDIA GPU metrics in Amazon CloudWatch , making it possible to push metrics from the Amazon CloudWatch Agent to Amazon CloudWatch and monitor your code for optimal GPU utilization. Then we explore two architectures. already installed.

Metrics 124

Metrics APIs Management Engineering 124

AWS Machine Learning

FEBRUARY 24, 2025

Performance metrics and benchmarks According to Mistral, the instruction-tuned version of the model achieves over 81% accuracy on Massive Multitask Language Understanding (MMLU) with 150 tokens per second latency, making it currently the most efficient model in its category. It doesnt support Converse APIs or other Amazon Bedrock tooling.

APIs 94

APIs Enterprise Benchmark Feedback 94

AWS Machine Learning

FEBRUARY 7, 2025

The implementation uses Slacks event subscription API to process incoming messages and Slacks Web API to send responses. The following screenshot shows an example. The incoming event from Slack is sent to an endpoint in API Gateway, and Slack expects a response in less than 3 seconds, otherwise the request fails.

Entertainment 85

Entertainment APIs Technology Feedback 85

AWS Machine Learning

NOVEMBER 15, 2024

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI.

Chatbots 113

Chatbots Engineering Enterprise Government 113

AWS Machine Learning

OCTOBER 24, 2024

Anatomy of RAG RAG is an efficient way to provide an FM with additional knowledge by using external data sources and is depicted in the following diagram: Retrieval : Based on a user’s question (1), relevant information is retrieved from a knowledge base (2) (for example, an OpenSearch index).

Chatbots 93

Chatbots Metrics Scripts APIs 93

AWS Machine Learning

NOVEMBER 19, 2024

Hear from customer speakers with real-world examples of how they’ve used data to support a variety of use cases, including generative AI, to create unique customer experiences. Discover how to create and manage evaluation jobs, use automatic and human reviews, and analyze critical metrics like accuracy, robustness, and toxicity.

APIs 107

APIs Enterprise Best practices Government 107

AWS Machine Learning

OCTOBER 16, 2024

Frontend and API The CQ application offers a robust search interface specially crafted for call quality agents, equipping them with powerful auditing capabilities for call analysis. Call Quality Trend Dashboard The following figure is an example of the Call Quality Trend Dashboard, showing the information available to agents.

Customer Service 88

Customer Service Scripts Coaching Finance 88

AWS Machine Learning

JANUARY 7, 2025

The translation playground could be adapted into a scalable serverless solution as represented by the following diagram using AWS Lambda , Amazon Simple Storage Service (Amazon S3), and Amazon API Gateway. For this example, the translated text, although accurate, is close to a literal translation, which is not a common phrasing in French.

Engineering 113

Engineering Metrics industry standards Analytics 113

AWS Machine Learning

JANUARY 25, 2023

For example, a fast food chain has launched its earn and burn loyalty pilot program in some locations. This post shows you how to use an integrated solution with Amazon Lookout for Metrics to break these barriers by quickly and easily detecting anomalies in the key performance indicators (KPIs) of your interest. Choose Upload.

Metrics 98

Metrics APIs Enterprise Analytics 98

AWS Machine Learning

APRIL 24, 2025

A seamless search journey not only enhances the overall user experience, but also directly impacts key business metrics such as conversion rates, average order value, and customer loyalty. Send the text, images, and metadata to Amazon Bedrock using its API to generate embeddings using the Amazon Titan Multimodal Embeddings G1 model.

Engineering 107

Engineering APIs Enterprise Accountability 107

AWS Machine Learning

AUGUST 7, 2024

For example, retrieving responses from its database before generating a response could provide more relevant and coherent responses. We then retrieve answers using standard RAG and a two-stage RAG, which involves a reranking API. The framework provides a suite of metrics to evaluate different dimensions. join(batch_text_arr) s3.put_object(

APIs 134

APIs Chatbots Metrics Education 134

AWS Machine Learning

MAY 7, 2024

Amazon Transcribe The transcription for the entire video is generated using the StartTranscriptionJob API. The raw transcripts are further processed to be stored using timestamps, as shown in the following example. The metadata generated for each video by the APIs is processed and stored with timestamps.

APIs 136

APIs Advertising Entertainment Metrics 136

AWS Machine Learning

OCTOBER 2, 2024

It’s a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies like Anthropic, Cohere, Meta, Mistral AI, and Amazon through a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI.

Government 129

Government APIs Enterprise Best practices 129

AWS Machine Learning

FEBRUARY 25, 2025

For example, What are the top sections of the HR benefits policies? Amazon Q Business only provides metric information that you can use to monitor your data source sync jobs. Create sample Alation policies In our example, you would create three different sets of Alation policies for a fictional organization named Unicorn Rentals.

Enterprise 105

Enterprise Engineering APIs Accountability 105

AWS Machine Learning

FEBRUARY 25, 2025

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon through a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI.

Chatbots 81

Chatbots Engineering Automotive Metrics 81