Generate high-quality meeting notes using Amazon Transcribe and Amazon Comprehend

One of the crucial challenges with online meetings is ensuring effective dissemination of details to all the participants after the meeting. You can get rid of such difficulties by using AWS artificial intelligence (AI) and machine learning (ML) technologies to create conference artifacts immediately, such as summaries, call-to-action products, and conference transcriptions.
In this post, we demonstrate a solution that utilizes the Amazon Chime SDK, Amazon Transcribe, Amazon Comprehend, and AWS Step Functions to record, process, and generate meeting artifacts. When the meeting bot stores the recorded file in an Amazon Simple Storage Service (Amazon S3) bucket, our proposed option is based on a Step Functions workflow that starts. The workflow consists of steps that transcribe and obtain insights from the meeting recording. It compiles the data into an e-mail design template and sends it to meeting guests. You can easily adapt this workflow for different use cases, such as web conferencing services.
Option introduction
The application is primarily divided into two parts: the conferencing service developed utilizing the Amazon Chime SDK, and the AI/ML-based processing workflow implemented utilizing Amazon Transcribe and Amazon Comprehend. The following diagram highlights the architecture.

Amazon Chime conferencing application
The conferencing application is a web-based application constructed using the Amazon Chime JS SDK and hosted using a mix of Amazon Elastic Container Service (Amazon ECS), AWS Lambda, and Amazon API Gateway. Session information for the meetings is saved in Amazon DynamoDB tables. During a conference call, the session info is recorded using an Amazon EventBridge connector for the Amazon Chime SDK, and written to the DynamoDB tables. The following features are offered online application:

Stop recording– When this action is initiated, it stops the Amazon ECS task running the headless web browser. Throughout the shutdown procedure, the Amazon ECS job composes the video recording into an S3 container.

Session metadata– For the period of the conference call, meeting metadata is streamed by an Amazon EventBridge Connector for Amazon Chime. The EventBridge rule is set up with a Lambda target and composes the information to a DynamoDB table.

Tape call– When the record call action is started, it starts an Amazon ECS job, which serves as the meeting recorder bot. This bot runs a headless Firefox web browser and signs up with the call as a participant. The headless internet browser is screen recorded in the Amazon ECS task utilizing FFMPEG and virtual audio routers.

Start or sign up with a call– When a user demands to begin a call or join, the demand conjures up the Amazon Chime SDK to start or join a conference. An unique MeetingId is created and passed along with the demand, and other individuals can use this MeetingId to join the same call.

The preceding functions allow users to start, participate in, and record conference calls. The call recording generates a video file that is delivered to an S3 container. The S3 container is configured with an Amazon S3 event notification for the s3: ObjectCreated: Put event, and initiates the AI/ML processing workflow These solutions are available as demos on the Amazon Chime JS SDK page on GitHub.
AI/ML processing workflow.
The AI/ML processing workflow built with Step Functions utilizes Amazon Transcribe and Amazon Comprehend. The output of this processing workflow is a well-crafted e-mail that is sent out to the teleconference owner using Amazon Simple Email Service (Amazon SES). The following series of steps is included in the AI/ML workflow:

Text, Type.
how to, QUESTIONS.
when can, QUESTIONS.
what is the, QUESTIONS.
schedule meeting, ACTIONS.
architecture, ACTIONS.
prices, ACTIONS.

,.
OutputDataConfig=
,.
EntityRecognizerArn= cer_arn, #The Amazon Resource Name (ARN) that identifies the specific entity recognizer.
LanguageCode= language_code, #Language code for the transcribed output.
DataAccessRoleArn= role,.
JobName= job_name, #Name of the task.
).

Output.
The following figure reveals a sample e-mail that is sent out to the meeting guests by the AI/ML processing workflow. The e-mail provides information such as the meeting title, participants, crucial discussion points, and the action products.

response = client.start _ entities_detection_job(.
InputDataConfig=
S 3Uri: input_path, #Location of the transcribed output.
InputFormat: ONE_DOC_PER_FILE #or ONE_DOC_PER_LINE.

The entire AI/ML processing workflow is displayed in the following figure.

In this post, we show a solution that utilizes the Amazon Chime SDK, Amazon Transcribe, Amazon Comprehend, and AWS Step Functions to tape, process, and produce conference artifacts. Our proposed service is based on a Step Functions workflow that begins when the meeting bot stores the taped file in an Amazon Simple Storage Service (Amazon S3) bucket. The conferencing application is a web-based application developed using the Amazon Chime JS SDK and hosted using a mix of Amazon Elastic Container Service (Amazon ECS), AWS Lambda, and Amazon API Gateway. The AI/ML processing workflow constructed with Step Functions utilizes Amazon Transcribe and Amazon Comprehend. The output of this processing workflow is a well-crafted email that is sent to the conference call owner utilizing Amazon Simple Email Service (Amazon SES).

The following is a sample code utilizing the Boto3 SDKs for beginning an asynchronous entity detection from the transcribed output:.

Speech to text– The place of the tape-recorded file in Amazon S3 is passed as a specification to the Amazon Transcribe start_transcription_job API that creates the asynchronous transcription task. Amazon Transcribe automatically transforms the taped speech to text precisely. If Amazon Transcribe needs to recognize domain-specific words and phrases such as product or brand name names, technical terminology, or names of individuals, there are 2 options: using the customized vocabulary function or utilizing custom language models.

Identify custom-made entities– After the transcribed text has been created, utilize the customized entity recognition function in Amazon Comprehend to draw out meeting highlights, follow-up actions, and concerns asked. You can likewise construct a customized entity recognizer using Amazon Comprehend. Amazon Comprehend will learn about the kind of documents and the context where the entities happen to develop the recognizer.

About the Authors.
Rajdeep Tarat is a Senior Solutions Architect at AWS. He resides in Bengaluru, India, and helps clients architect and enhance applications on AWS. In his spare time, he takes pleasure in music, shows, and reading.
Venugopal Pai is a Solutions Architect at AWS. He lives in Bengaluru, India, and helps digital native clients scale and optimize their applications on AWS.

response = client.start _ transcription_job(.
TranscriptionJobName= job_name, #Name of the task.
LanguageCode= language_code, #Language code for the language in media file.
MediaFormat= media_format, #Format of input media file.
Media=
MediaFileUri: file_uri #S 3 item location of input media file.
,.
Settings=
VocabularyName: vocab_name #Name of the customized vocabulary to use.
).

Send out e-mail– The processing from the previous step produces data that is saved in an S3 pail and a DynamoDB table. These outcomes are collected by a Lambda function, formatted into an e-mail, and sent out across to the meeting guests utilizing Amazon SES.

Summary.
In this post, we demonstrated how you can use AWS AI services such as Amazon Transcribe and Amazon Comprehend together with the Amazon Chime SDK to generate high-quality meeting artifacts. We showed the custom-made vocabulary function of Amazon Transcribe and the custom-made entities feature of Amazon Comprehend that allow you to customize the artifacts based on your company requirements.
Discover more about AWS AI services and begin constructing your own customized processing workflow utilizing AWS Step Functions and Amazon Chime SDK.

Leave a Reply

Your email address will not be published.