AWS Machine Learning

FEBRUARY 12, 2025

This approach allows organizations to assess their AI models effectiveness using pre-defined metrics, making sure that the technology aligns with their specific needs and objectives. The introduction of an LLM-as-a-judge framework represents a significant step forward in simplifying and streamlining the model evaluation process.

Metrics 109

Metrics Engineering Benchmark APIs 109

AWS Machine Learning

OCTOBER 29, 2024

All of this data is centralized and can be used to improve metrics in scenarios such as sales or call centers. For integration between services, we use API Gateway as an event trigger for our Lambda function, and DynamoDB as a highly scalable database to store our customer details.

Engineering 112

Engineering APIs Sales Enterprise 112

AWS Machine Learning

DECEMBER 2, 2024

Use faster auto scaling metrics – Take advantage of more granular auto scaling metrics like ConcurrentRequestsPerCopy to more accurately monitor and react to changes in inference traffic. It’s a dynamic policy that adjusts the number of copies based on a specified metric, such as CPU utilization or request count.

APIs 109

APIs Best practices Metrics Engineering 109

AWS Machine Learning

MARCH 6, 2025

Current RAG pipelines frequently employ similarity-based metrics such as ROUGE , BLEU , and BERTScore to assess the quality of the generated responses, which is essential for refining and enhancing the models capabilities. More sophisticated metrics are needed to evaluate factual alignment and accuracy.

Metrics 120

Metrics Enterprise APIs Engineering 120

AWS Machine Learning

AUGUST 2, 2024

adds new APIs to customize GraphStorm pipelines: you now only need 12 lines of code to implement a custom node classification training loop. Based on customer feedback for the experimental APIs we released in GraphStorm 0.2, introduces refactored graph ML pipeline APIs. Specifically, GraphStorm 0.3 In addition, GraphStorm 0.3

APIs 118

APIs Benchmark Construction Enterprise 118

AWS Machine Learning

OCTOBER 29, 2024

Observability refers to the ability to understand the internal state and behavior of a system by analyzing its outputs, logs, and metrics. Evaluation, on the other hand, involves assessing the quality and relevance of the generated outputs, enabling continual improvement.

Best practices 114

Best practices Feedback Metrics APIs 114

AWS Machine Learning

DECEMBER 9, 2024

However, keeping track of numerous experiments, their parameters, metrics, and results can be difficult, especially when working on complex projects simultaneously. Note that MLflow tracking starts from the mlflow.start_run() API. The mlflow.autolog() API can automatically log information such as metrics, parameters, and artifacts.

Metrics 106

Metrics APIs Engineering Accountability 106

AWS Machine Learning

APRIL 10, 2025

Fine-tune an Amazon Nova model using the Amazon Bedrock API In this section, we provide detailed walkthroughs on fine-tuning and hosting customized Amazon Nova models using Amazon Bedrock. We first provided a detailed walkthrough on how to fine-tune, host, and conduct inference with customized Amazon Nova through the Amazon Bedrock API.

APIs 125

APIs Metrics Best practices Construction 125

AWS Machine Learning

MARCH 25, 2025

Automated safety guards Integrated Amazon CloudWatch alarms monitor metrics on an inference component. AlarmName This CloudWatch alarm is configured to monitor metrics on an InferenceComponent. For more information about the SageMaker AI API, refer to the SageMaker AI API Reference.

APIs 111

APIs Engineering Accountability Metrics 111

AWS Machine Learning

OCTOBER 9, 2024

Amazon Lookout for Metrics is a fully managed service that uses machine learning (ML) to detect anomalies in virtually any time-series business or operational metrics—such as revenue performance, purchase transactions, and customer acquisition and retention rates—with no ML experience required.

Metrics 95

Metrics APIs Engineering Accountability 95

AWS Machine Learning

SEPTEMBER 28, 2023

In this post, we describe the enhancements to the forecasting capabilities of SageMaker Canvas and guide you on using its user interface (UI) and AutoML APIs for time-series forecasting. While the SageMaker Canvas UI offers a code-free visual interface, the APIs empower developers to interact with these features programmatically.

APIs 120

APIs Construction Finance Enterprise 120

AWS Machine Learning

JANUARY 28, 2025

To effectively optimize AI applications for responsiveness, we need to understand the key metrics that define latency and how they impact user experience. These metrics differ between streaming and nonstreaming modes and understanding them is crucial for building responsive AI applications.

Benchmark 80

Benchmark APIs Engineering Metrics 80

AWS Machine Learning

SEPTEMBER 7, 2023

In February 2022, Amazon Web Services added support for NVIDIA GPU metrics in Amazon CloudWatch , making it possible to push metrics from the Amazon CloudWatch Agent to Amazon CloudWatch and monitor your code for optimal GPU utilization. Then we explore two architectures. already installed. eks-create.sh 19 private:192.168.128.0/19

Metrics 125

Metrics APIs Management Engineering 125

AWS Machine Learning

APRIL 10, 2025

For instance, Pixtral Large is highly effective at spotting irregularities or insightful trends within training loss curves or performance metrics, enhancing the accuracy of data-driven decision-making. By choosing View API , you can also access the model using code examples in the AWS Command Line Interface (AWS CLI) and AWS SDKs.

APIs 112

APIs Management Marketing Engineering 112

AWS Machine Learning

SEPTEMBER 20, 2024

Amazon Bedrock agents use LLMs to break down tasks, interact dynamically with users, run actions through API calls, and augment knowledge using Amazon Bedrock Knowledge Bases. In this post, we demonstrate how to use Amazon Bedrock Agents with a web search API to integrate dynamic web content in your generative AI application.

APIs 106

APIs Chatbots Construction Engineering 106

AWS Machine Learning

FEBRUARY 18, 2025

During these live events, F1 IT engineers must triage critical issues across its services, such as network degradation to one of its APIs. This impacts downstream services that consume data from the API, including products such as F1 TV, which offer live and on-demand coverage of every race as well as real-time telemetry.

APIs 72

APIs Engineering Metrics Big data 72

AWS Machine Learning

FEBRUARY 21, 2025

The solution uses the FMs tool use capabilities, accessed through the Amazon Bedrock Converse API. This enables the FMs to not just process text, but to actively engage with various external tools and APIs to perform complex document analysis tasks. For more details on how tool use works, refer to The complete tool use workflow.

APIs 98

APIs Healthcare Consulting Consulting 98

AWS Machine Learning

NOVEMBER 20, 2024

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI.

Construction 118

Construction APIs Metrics Accountability 118

AWS Machine Learning

MARCH 3, 2025

Performance metrics and benchmarks Pixtral 12B is trained to understand both natural images and documents, achieving 52.5% You can find detailed usage instructions, including sample API calls and code snippets for integration. To begin using Pixtral 12B, choose Deploy.

Benchmark 113

Benchmark APIs Enterprise Construction 113

AWS Machine Learning

MARCH 14, 2025

Although automated metrics are fast and cost-effective, they can only evaluate the correctness of an AI response, without capturing other evaluation dimensions or providing explanations of why an answer is problematic. Human evaluation, although thorough, is time-consuming and expensive at scale.

Metrics 109

Metrics Best practices Engineering APIs 109

AWS Machine Learning

MARCH 27, 2025

Where discrete outcomes with labeled data exist, standard ML methods such as precision, recall, or other classic ML metrics can be used. These metrics provide high precision but are limited to specific use cases due to limited ground truth data. If the use case doesnt yield discrete outputs, task-specific metrics are more appropriate.

Education 112

Education Engineering APIs Enterprise 112

AWS Machine Learning

NOVEMBER 19, 2024

Gain insights into training strategies, productivity metrics, and real-world use cases to empower your developers to harness the full potential of this game-changing technology. Discover how to create and manage evaluation jobs, use automatic and human reviews, and analyze critical metrics like accuracy, robustness, and toxicity.

APIs 110

APIs Enterprise Best practices Government 110

AWS Machine Learning

NOVEMBER 13, 2024

The Amazon Bedrock single API access, regardless of the models you choose, gives you the flexibility to use different FMs and upgrade to the latest model versions with minimal code changes. Amazon Titan FMs provide customers with a breadth of high-performing image, multimodal, and text model choices, through a fully managed API.

Engineering 116

Engineering Management APIs Healthcare 116

AWS Machine Learning

FEBRUARY 24, 2025

Performance metrics and benchmarks According to Mistral, the instruction-tuned version of the model achieves over 81% accuracy on Massive Multitask Language Understanding (MMLU) with 150 tokens per second latency, making it currently the most efficient model in its category. It doesnt support Converse APIs or other Amazon Bedrock tooling.

APIs 99

APIs Enterprise Benchmark Feedback 99

AWS Machine Learning

NOVEMBER 14, 2024

The user’s request is sent to AWS API Gateway , which triggers a Lambda function to interact with Amazon Bedrock using Anthropic’s Claude Instant V1 FM to process the user’s request and generate a natural language response of the place location. It will then return the place name with the highest similarity score.

APIs 112

APIs Engineering Personalization Transportation 112

AWS Machine Learning

NOVEMBER 15, 2024

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon through a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI.

Chatbots 114

Chatbots Engineering Enterprise Government 114

AWS Machine Learning

OCTOBER 2, 2024

It’s a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies like Anthropic, Cohere, Meta, Mistral AI, and Amazon through a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI.

Government 130

Government APIs Enterprise Best practices 130

AWS Machine Learning

AUGUST 7, 2024

We then retrieve answers using standard RAG and a two-stage RAG, which involves a reranking API. Retrieve answers using the knowledge base retrieve API Evaluate the response using the RAGAS Retrieve answers again by running a two-stage RAG, using the knowledge base retrieve API and then applying reranking on the context.

APIs 135

APIs Chatbots Metrics Education 135

AWS Machine Learning

FEBRUARY 7, 2025

The implementation uses Slacks event subscription API to process incoming messages and Slacks Web API to send responses. The incoming event from Slack is sent to an endpoint in API Gateway, and Slack expects a response in less than 3 seconds, otherwise the request fails.

Entertainment 90

Entertainment APIs Technology Feedback 90

AWS Machine Learning

MAY 7, 2024

Amazon Transcribe The transcription for the entire video is generated using the StartTranscriptionJob API. The solution runs Amazon Rekognition APIs for label detection , text detection, celebrity detection , and face detection on videos. The metadata generated for each video by the APIs is processed and stored with timestamps.

APIs 137

APIs Advertising Entertainment Metrics 137

AWS Machine Learning

MARCH 27, 2024

This requires carefully combining applications and metrics to provide complete awareness, accuracy, and control. The zAdviser uses Amazon Bedrock to provide summarization, analysis, and recommendations for improvement based on the DORA metrics data. It’s also vital to avoid focusing on irrelevant metrics or excessively tracking data.

Enterprise 122

Enterprise APIs Metrics Engineering 122

AWS Machine Learning

APRIL 24, 2025

A seamless search journey not only enhances the overall user experience, but also directly impacts key business metrics such as conversion rates, average order value, and customer loyalty. Send the text, images, and metadata to Amazon Bedrock using its API to generate embeddings using the Amazon Titan Multimodal Embeddings G1 model.

Engineering 108

Engineering APIs Enterprise Accountability 108

AWS Machine Learning

OCTOBER 16, 2024

Frontend and API The CQ application offers a robust search interface specially crafted for call quality agents, equipping them with powerful auditing capabilities for call analysis. Additionally, the application offers backend dashboards tailored to MLOps functionalities, ensuring smooth monitoring and optimization of machine learning models.

Customer Service 89

Customer Service Scripts Coaching Finance 89

AWS Machine Learning

MAY 30, 2024

AWS Prototyping successfully delivered a scalable prototype, which solved CBRE’s business problem with a high accuracy rate (over 95%) and supported reuse of embeddings for similar NLQs, and an API gateway for integration into CBRE’s dashboards. The following diagram illustrates the web interface and API management layer.

Real estate 130

Real estate APIs Metrics Construction 130

AWS Machine Learning

JANUARY 10, 2024

The solution uses AWS Lambda , Amazon API Gateway , Amazon EventBridge , and SageMaker to automate the workflow with human approval intervention in the middle. The approver approves the model by following the link in the email to an API Gateway endpoint. API Gateway invokes a Lambda function to initiate model updates.

APIs 130

APIs Construction Engineering Analytics 130

AWS Machine Learning

OCTOBER 24, 2024

The retrieve_and_generate API does both the retrieval and a call to an FM (Amazon Titan or Anthropic’s Claude family of models on Amazon Bedrock ), for a fully managed solution. Mean Reciprocal Rank (MRR) – This metric considers the ranking of the retrieved documents. More advanced models such as Anthropic’s Claude Sonnet 3.5

Chatbots 94

Chatbots Metrics Scripts APIs 94

AWS Machine Learning

AUGUST 26, 2024

The GenASL web app invokes the backend services by sending the S3 object key in the payload to an API hosted on Amazon API Gateway. API Gateway instantiates an AWS Step Functions The state machine orchestrates the AI/ML services Amazon Transcribe and Amazon Bedrock and the NoSQL data store Amazon DynamoDB using AWS Lambda functions.

APIs 122

APIs Calibration Best practices Metrics 122