APIs, Benchmark and Examples - Customer Contact Central

Benchmarking Amazon Nova and GPT-4o models with FloTorch

AWS Machine Learning

MARCH 11, 2025

Using its enterprise software, FloTorch conducted an extensive comparison between Amazon Nova models and OpenAIs GPT-4o models with the Comprehensive Retrieval Augmented Generation (CRAG) benchmark dataset. The following table provides example questions with their domain and question type.

Benchmark

Benchmark APIs Enterprise Scripts

GraphStorm 0.3: Scalable, multi-task learning on graphs with user-friendly APIs

AWS Machine Learning

AUGUST 2, 2024

adds new APIs to customize GraphStorm pipelines: you now only need 12 lines of code to implement a custom node classification training loop. To help you get started with the new API, we have published two Jupyter notebook examples: one for node classification, and one for a link prediction task. Specifically, GraphStorm 0.3

APIs

APIs Benchmark Construction Enterprise

Pixtral-12B-2409 is now available on Amazon Bedrock Marketplace

AWS Machine Learning

MARCH 3, 2025

Overview of Pixtral 12B Pixtral 12B, Mistrals inaugural VLM, delivers robust performance across a range of benchmarks, surpassing other open models and rivaling larger counterparts, according to Mistrals evaluation. Performance metrics and benchmarks Pixtral 12B is trained to understand both natural images and documents, achieving 52.5%

Benchmark

Benchmark APIs Enterprise Construction

Anthropic Claude 3.5 Sonnet ranks number 1 for business and finance in S&P AI Benchmarks by Kensho

AWS Machine Learning

JULY 9, 2024

Sonnet currently ranks at the top of S&P AI Benchmarks by Kensho , which assesses large language models (LLMs) for finance and business. For example, there could be leakage of benchmark datasets’ questions and answers into training data. Anthropic Claude 3.5 Kensho is the AI Innovation Hub for S&P Global.

Finance

Finance Benchmark industry standards Accountability

Optimizing AI responsiveness: A practical guide to Amazon Bedrock latency-optimized inference

AWS Machine Learning

JANUARY 28, 2025

For example, a prompt that generates 100 tokens in one model might generate 150 tokens in another. Consider benchmarking your user experience to find the best latency for your use case, considering that most humans cant read faster than 225 words per minute and therefore extremely fast response can hinder user experience.

Benchmark

Benchmark APIs Engineering Metrics

LLM-as-a-judge on Amazon Bedrock Model Evaluation

AWS Machine Learning

FEBRUARY 12, 2025

Amazon Bedrock , a fully managed service offering high-performing foundation models from leading AI companies through a single API, has recently introduced two significant evaluation capabilities: LLM-as-a-judge under Amazon Bedrock Model Evaluation and RAG evaluation for Amazon Bedrock Knowledge Bases. 0]}-{evaluator_model.split('.')[0]}-{datetime.now().strftime('%Y-%m-%d-%H-%M-%S')}"

Metrics

Metrics Engineering APIs Benchmark

Benchmark and optimize endpoint deployment in Amazon SageMaker JumpStart

AWS Machine Learning

JANUARY 29, 2024

This post explores these relationships via a comprehensive benchmarking of LLMs available in Amazon SageMaker JumpStart, including Llama 2, Falcon, and Mistral variants. We provide theoretical principles on how accelerator specifications impact LLM benchmarking. Additionally, models are fully sharded on the supported instance.

Benchmark

Benchmark APIs Enterprise Accountability

Generate training data and cost-effectively train categorical models with Amazon Bedrock

AWS Machine Learning

MARCH 27, 2025

We also showcase a real-world example for predicting the root cause category for support cases. For the use case of labeling the support root cause categories, its often harder to source examples for categories such as Software Defect, Feature Request, and Documentation Improvement for labeling than it is for Customer Education.

Education

Education Engineering APIs Enterprise

From RAG to fabric: Lessons learned from building real-world RAGs at GenAIIC – Part 2

AWS Machine Learning

NOVEMBER 15, 2024

For example, a technician could query the system about a specific machine part, receiving both textual maintenance history and annotated images showing wear patterns or common failure points, enhancing their ability to diagnose and resolve issues efficiently. has 92% accuracy on the HumanEval code benchmark.

APIs

APIs Engineering Chatbots Construction

Evaluate RAG responses with Amazon Bedrock, LlamaIndex and RAGAS

AWS Machine Learning

MARCH 6, 2025

Here are some examples of these metrics: Retrieval component Context precision Evaluates whether all of the ground-truth relevant items present in the contexts are ranked higher or not. For example, metrics like Answer Relevancy and Faithfulness are typically scored on a scale from 0 to 1.

Metrics

Metrics Enterprise APIs Engineering

Fine-tune LLMs with synthetic data for context-based Q&A using Amazon Bedrock

AWS Machine Learning

FEBRUARY 12, 2025

Amazon Bedrock is a fully managed service that makes FMs from leading AI startups and Amazon available through an API, so you can choose from a wide range of FMs to find the model that is best suited for your use case. Solution overview The solution comprises two main steps: Generate synthetic data using the Amazon Bedrock InvokeModel API.

APIs

APIs Management Benchmark Scripts

Mistral-Small-24B-Instruct-2501 is now available on SageMaker Jumpstart and Amazon Bedrock Marketplace

AWS Machine Learning

FEBRUARY 24, 2025

Performance metrics and benchmarks According to Mistral, the instruction-tuned version of the model achieves over 81% accuracy on Massive Multitask Language Understanding (MMLU) with 150 tokens per second latency, making it currently the most efficient model in its category. It doesnt support Converse APIs or other Amazon Bedrock tooling.

APIs

APIs Enterprise Benchmark Feedback

Get started with Amazon Titan Text Embeddings V2: A new state-of-the-art embeddings model on Amazon Bedrock

AWS Machine Learning

MAY 2, 2024

We published a follow-up post on January 31, 2024, and provided code examples using AWS SDKs and LangChain, showcasing a Streamlit semantic search app. For example, in a recommendation system for a large ecommerce platform, a modest increase in recommendation accuracy could translate into significant additional revenue.

Benchmark

Benchmark Metrics Enterprise APIs

Your guide to generative AI and ML at AWS re:Invent 2024

AWS Machine Learning

NOVEMBER 19, 2024

Hear from customer speakers with real-world examples of how they’ve used data to support a variety of use cases, including generative AI, to create unique customer experiences. Explore examples to estimate the severity and likelihood of potential events that could be harmful.

APIs

APIs Enterprise Best practices Government

Intelligent healthcare forms analysis with Amazon Bedrock

AWS Machine Learning

AUGUST 13, 2024

Amazon Bedrock is a fully managed service that makes foundation models (FMs) from leading AI startups and Amazon available through an API, so you can choose from a wide range of FMs to find the model that is best suited for your use case. Lastly, the Lambda function stores the question list in Amazon S3.

Healthcare

Healthcare APIs Consulting Consulting

Package and deploy classical ML and LLMs easily with Amazon SageMaker, part 2: Interactive User Experiences in SageMaker Studio

AWS Machine Learning

NOVEMBER 30, 2023

SageMaker makes it easy to deploy models into production directly through API calls to the service. It’s a low-level API available for Java, C++, Go, JavaScript, Node.js, PHP, Ruby, and Python. It’s a low-level API available for Java, C++, Go, JavaScript, Node.js, PHP, Ruby, and Python.

Benchmark

Benchmark APIs Scripts Engineering

AI21 Labs Jamba-Instruct model is now available in Amazon Bedrock

AWS Machine Learning

JUNE 25, 2024

Example use cases for Jamba-Instruct Jamba-Instruct’s long context length is particularly well-suited for complex Retrieval Augmented Generation (RAG) workloads, or potentially complex document analysis. For example: You are a curious and novel researcher, who is highly interested in getting all the relevant information on a specific topic.

APIs

APIs Benchmark Enterprise Technology

Learn how Amazon Ads created a generative AI-powered image generation capability using Amazon SageMaker

AWS Machine Learning

MAY 15, 2024

For example, an advertiser might have static images of their product against a white background. An example from the image generation solution showing a hydro flask with various backgrounds. Acting as a model hub, JumpStart provided a large selection of foundation models and the team quickly ran their benchmarks on candidate models.

Advertising

Advertising APIs Engineering Benchmark

Reduce conversational AI response time through inference at the edge with AWS Local Zones

AWS Machine Learning

MARCH 3, 2025

They enable applications requiring very low latency or local data processing using familiar APIs and tool sets. Through comparative benchmarking tests, we illustrate how deploying FMs in Local Zones closer to end users can significantly reduce latencya critical factor for real-time applications such as conversational AI assistants.

APIs

APIs Benchmark Metrics Healthcare

Build a secure enterprise application with Generative AI and RAG using Amazon SageMaker JumpStart

AWS Machine Learning

SEPTEMBER 6, 2023

These SageMaker endpoints are consumed in the Amplify React application through Amazon API Gateway and AWS Lambda functions. To protect the application and APIs from inadvertent access, Amazon Cognito is integrated into Amplify React, API Gateway, and Lambda functions. For this example, we use train.cc_casebooks.jsonl.xz

Enterprise

Enterprise APIs Real estate Construction

Accelerated PyTorch inference with torch.compile on AWS Graviton processors

AWS Machine Learning

JULY 2, 2024

You can see that for the 45 models we benchmarked, there is a 1.35x latency improvement (geomean for the 45 models). You can see that for the 33 models we benchmarked, there is around 2x performance improvement (geomean for the 33 models). We benchmarked 45 models using the scripts from the TorchBench repo.

Benchmark

Benchmark Scripts Metrics APIs

Optimize your machine learning deployments with auto scaling on Amazon SageMaker

AWS Machine Learning

FEBRUARY 8, 2023

Although you can integrate the model directly into an application, the approach that works well for production-grade applications is to deploy the model behind an endpoint and then invoke the endpoint via a RESTful API call to obtain the inference. However, you can use any other benchmarking tool. This value range is 0–100%.

Benchmark

Benchmark Metrics APIs Engineering

Evaluation of generative AI techniques for clinical report summarization

AWS Machine Learning

MAY 13, 2024

This is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading artificial intelligence (AI) companies like AI21 Labs, Anthropic, Cohere, Meta, Stability AI, and Amazon through a single API. A prompt has three main types of content: input, context, and examples.

Healthcare

Healthcare Engineering APIs Benchmark

How to use public reviews to improve customer experience

Lumoa

DECEMBER 8, 2022

How to add your competitors’ public reviews to the mix, so you can get a reliable benchmark and win in the market. There are many ways to use the integration function, and this is just one example. Lumoa has very clear open APIs and based on your goals, integrations can serve various purposes.

Customer Experience

Customer Experience APIs Benchmark Marketing

The executive’s guide to generative AI for sustainability

AWS Machine Learning

APRIL 22, 2024

It provides examples of use cases and best practices for using generative AI’s potential to accelerate sustainability and ESG initiatives, as well as insights into the main operational challenges of generative AI for sustainability. Figure 1 illustrates selected examples of use cases of generative AI for sustainability across the value chain.

Best practices

Best practices Benchmark Transportation Engineering

Exciting new developments from Spearline

Spearline

FEBRUARY 3, 2020

With such a rise in popularity of mobile usage around the world, we are delighted to announce that from February 2020, our customers will be able to test the sending of an SMS message to a destination specified by them, via the Spearline API. Access real-time reporting and analytics via Spearline API polling.

Telecommunications

Telecommunications APIs Benchmark Enterprise

Build a multilingual automatic translation pipeline with Amazon Translate Active Custom Translation

AWS Machine Learning

JUNE 15, 2023

It features interactive Jupyter notebooks with self-contained code in PyTorch, JAX, TensorFlow, and MXNet, as well as real-world examples, exposition figures, and math. ACT allows you to customize translation output on the fly by providing tailored translation examples in the form of parallel data.

APIs

APIs Benchmark Best practices Engineering

Optimize pet profiles for Purina’s Petfinder application using Amazon Rekognition Custom Labels and AWS Step Functions

AWS Machine Learning

OCTOBER 18, 2023

The solution uses the following services: Amazon API Gateway is a fully managed service that makes it easy for developers to publish, maintain, monitor, and secure APIs at any scale. Purina’s solution is deployed as an API Gateway HTTP endpoint, which routes the requests to obtain pet attributes.

APIs

APIs Metrics Consulting Consulting

A review of purpose-built accelerators for financial services

AWS Machine Learning

SEPTEMBER 11, 2024

In this example figure, features are extracted from raw historical data, which are then are fed into a neural network (NN). In terms of resulting speedups, the approximate order is programming hardware, then programming against PBA APIs, then programming in an unmanaged language such as C++, then a managed language such as Python.

Benchmark

Benchmark Banking Analytics Big data

A progress update on our commitment to safe, responsible generative AI

AWS Machine Learning

JULY 10, 2024

Red-teaming engages human testers to probe an AI system for flaws in an adversarial style, and complements our other testing techniques, which include automated benchmarking against publicly available and proprietary datasets, human evaluation of completions against proprietary datasets, and more.

Government

Government Education Best practices APIs

Build a RAG-based QnA application using Llama3 models from SageMaker JumpStart

AWS Machine Learning

SEPTEMBER 12, 2024

On Hugging Face, the Massive Text Embedding Benchmark (MTEB) is provided as a leaderboard for diverse text embedding tasks. It currently provides 129 benchmarking datasets across 8 different tasks on 113 languages. medium instance to demonstrate deploying the model as an API endpoint using an SDK through SageMaker JumpStart.

APIs

APIs Benchmark Enterprise Construction

Improved ML model deployment using Amazon SageMaker Inference Recommender

AWS Machine Learning

APRIL 20, 2023

The majority of data is non-fraudulent (284,315 samples), with only 492 samples corresponding to fraudulent examples. An advanced job is a custom load test job that allows you to perform extensive benchmarks based on your ML application SLA requirements, such as latency, concurrency, and traffic pattern. Running Advanced job.

APIs

APIs Metrics Benchmark Engineering

Image classification model selection using Amazon SageMaker JumpStart

AWS Machine Learning

FEBRUARY 6, 2023

The former question addresses model selection across model architectures, while the latter question concerns benchmarking trained models against a test dataset. This post provides details on how to implement large-scale Amazon SageMaker benchmarking and model selection tasks. swin-large-patch4-window7-224 195.4M efficientnet-b5 29.0M

APIs

APIs Scripts Metrics Benchmark

Gemma is now available in Amazon SageMaker JumpStart

AWS Machine Learning

MARCH 13, 2024

You will also find a Deploy button, which takes you to a landing page where you can test inference with an example payload. Deploy Gemma with SageMaker Python SDK You can find the code showing the deployment of Gemma on JumpStart and an example of how to use the deployed model in this GitHub notebook. This looks pretty good!

Benchmark

Benchmark Scripts APIs Feedback

Snowflake Arctic models are now available in Amazon SageMaker JumpStart

AWS Machine Learning

AUGUST 22, 2024

Snowflake Arctic is a family of enterprise-grade large language models (LLMs) built by Snowflake to cater to the needs of enterprise users, exhibiting exceptional capabilities (as shown in the following benchmarks ) in SQL querying, coding, and accurately following instructions. To learn more, refer to API documentation.

Enterprise

Enterprise APIs Benchmark Scripts

Run PyTorch Lightning and native PyTorch DDP on Amazon SageMaker Training, featuring Amazon Search

AWS Machine Learning

AUGUST 18, 2022

The team’s early benchmarking results show 7.3 The baseline model used in these benchmarking is a multi-layer perceptron neural network with seven dense fully connected layers and over 200 parameters. The following table summarizes the benchmarking result on ml.p3.16xlarge SageMaker training instances. Number of Instances.

Scripts

Scripts APIs Benchmark Engineering

Best practices for building robust generative AI applications with Amazon Bedrock Agents – Part 1

AWS Machine Learning

OCTOBER 2, 2024

In addition, they use the developer-provided instruction to create an orchestration plan and then carry out the plan by invoking company APIs and accessing knowledge bases using Retrieval Augmented Generation (RAG) to provide an answer to the user’s request. In Part 1, we focus on creating accurate and reliable agents.

Best practices

Best practices APIs Metrics Accountability

The 13 Best AI Chatbots for Business in 2021 and Beyond [Review and Key Features]

Netomi

SEPTEMBER 20, 2021

We’ll also cover the 5 best chatbot examples in the real world, but more on that later. CSML helps developers build and deploy chatbots easily with its expressive syntax and its capacity to connect to any third party API. Self-service APIs to help you create, manage, test and publish custom skills. Key features: .

Chatbots

Chatbots APIs Surveys Analytics

Build RAG applications using Jina Embeddings v2 on Amazon SageMaker JumpStart

AWS Machine Learning

JUNE 6, 2024

Jina Embeddings v2 is the preferred choice for experienced ML scientists for the following reasons: State-of-the-art performance – We have shown on various text embedding benchmarks that Jina Embeddings v2 models excel on tasks such as classification, reranking, summarization, and retrieval.

Benchmark

Benchmark Enterprise Construction APIs

Evaluate conversational AI agents with Amazon Bedrock

AWS Machine Learning

JULY 25, 2024

Although existing large language model (LLM) benchmarks like MT-bench evaluate model capabilities, they lack the ability to validate the application layers. In the following example test plan, you have a claim with the ID claim-006 in your test system. Configure the target, which should be a type that Agent Evaluation supports.

APIs

APIs Engineering Best practices Virtual Agent

Scalable intelligent document processing using Amazon Bedrock

AWS Machine Learning

JUNE 12, 2024

Amazon Bedrock is a fully managed service that makes foundation models (FMs) from leading artificial intelligence (AI) startups and Amazon available through an API, so you can choose from a wide range of FMs to find the model that is best suited for your use case. The following screenshot shows an example email.

APIs

APIs Accountability Benchmark Government

How Forethought saves over 66% in costs for generative AI models using Amazon SageMaker

AWS Machine Learning

JUNE 13, 2023

For example, in order to share expensive GPU resources between multiple models, we were responsible for allocating rigid memory fractions to models that were specified during deployment. In addition, deployments are now as simple as calling Boto3 SageMaker APIs and attaching the proper auto scaling policies.

APIs

APIs Benchmark Engineering Management

Best practices for load testing Amazon SageMaker real-time inference endpoints

AWS Machine Learning

JANUARY 10, 2023

We first benchmark the performance of our model on a single instance to identify the TPS it can handle per our acceptable latency requirements. The entire set of code for the example is available in the following GitHub repository. Overview of solution. For CPUUtilization , you may see percentages above 100% at first in CloudWatch.

Best practices

Best practices Scripts APIs Metrics

Build well-architected IDP solutions with a custom lens – Part 4: Performance efficiency

AWS Machine Learning

NOVEMBER 22, 2023

You can save time, money, and labor by implementing classifications in your workflow, and documents go to downstream applications and APIs based on document type. For example, by storing a document in the different processed phases, you can reverse processing to the previous step if needed.

APIs

APIs Metrics Benchmark Enterprise

Benchmarking Amazon Nova and GPT-4o models with FloTorch

GraphStorm 0.3: Scalable, multi-task learning on graphs with user-friendly APIs

Trending Sources

Pixtral-12B-2409 is now available on Amazon Bedrock Marketplace

Anthropic Claude 3.5 Sonnet ranks number 1 for business and finance in S&P AI Benchmarks by Kensho

Optimizing AI responsiveness: A practical guide to Amazon Bedrock latency-optimized inference

LLM-as-a-judge on Amazon Bedrock Model Evaluation

Benchmark and optimize endpoint deployment in Amazon SageMaker JumpStart

Generate training data and cost-effectively train categorical models with Amazon Bedrock

From RAG to fabric: Lessons learned from building real-world RAGs at GenAIIC – Part 2

Evaluate RAG responses with Amazon Bedrock, LlamaIndex and RAGAS

Fine-tune LLMs with synthetic data for context-based Q&A using Amazon Bedrock

Mistral-Small-24B-Instruct-2501 is now available on SageMaker Jumpstart and Amazon Bedrock Marketplace

Get started with Amazon Titan Text Embeddings V2: A new state-of-the-art embeddings model on Amazon Bedrock

Your guide to generative AI and ML at AWS re:Invent 2024

Intelligent healthcare forms analysis with Amazon Bedrock

Package and deploy classical ML and LLMs easily with Amazon SageMaker, part 2: Interactive User Experiences in SageMaker Studio

AI21 Labs Jamba-Instruct model is now available in Amazon Bedrock

Learn how Amazon Ads created a generative AI-powered image generation capability using Amazon SageMaker

Reduce conversational AI response time through inference at the edge with AWS Local Zones

Build a secure enterprise application with Generative AI and RAG using Amazon SageMaker JumpStart

Accelerated PyTorch inference with torch.compile on AWS Graviton processors

Optimize your machine learning deployments with auto scaling on Amazon SageMaker

Evaluation of generative AI techniques for clinical report summarization

How to use public reviews to improve customer experience

The executive’s guide to generative AI for sustainability

Exciting new developments from Spearline

Build a multilingual automatic translation pipeline with Amazon Translate Active Custom Translation

Optimize pet profiles for Purina’s Petfinder application using Amazon Rekognition Custom Labels and AWS Step Functions

A review of purpose-built accelerators for financial services

A progress update on our commitment to safe, responsible generative AI

Build a RAG-based QnA application using Llama3 models from SageMaker JumpStart

Improved ML model deployment using Amazon SageMaker Inference Recommender

Image classification model selection using Amazon SageMaker JumpStart

Gemma is now available in Amazon SageMaker JumpStart

Snowflake Arctic models are now available in Amazon SageMaker JumpStart

Run PyTorch Lightning and native PyTorch DDP on Amazon SageMaker Training, featuring Amazon Search

Best practices for building robust generative AI applications with Amazon Bedrock Agents – Part 1

The 13 Best AI Chatbots for Business in 2021 and Beyond [Review and Key Features]

Build RAG applications using Jina Embeddings v2 on Amazon SageMaker JumpStart

Evaluate conversational AI agents with Amazon Bedrock

Scalable intelligent document processing using Amazon Bedrock

How Forethought saves over 66% in costs for generative AI models using Amazon SageMaker

Best practices for load testing Amazon SageMaker real-time inference endpoints

Build well-architected IDP solutions with a custom lens – Part 4: Performance efficiency

Stay Connected