AI-Powered Meeting Assistant for Transcription, Summarization and Action Extraction by IRJET Journal

AI-Powered Meeting Assistant for Transcription, Summarization and Action Extraction

Dr. Rupali Mishra

Coordinator, Department of Information Technology, Prahadrai Dalmia Lions College, Mumbai, India

Abstract - Meetings are an integral part of organizational communication, yet documenting them effectively remains a persistent challenge. Traditional manual note-taking is timeconsuming, error-prone, and often results in incomplete information capture. This paper presents AutoScribe, an AIpowered meeting assistant designed to automate transcription, summarization,andactionitemextractionfrom meeting recordings. The system leverages state-of-the-art speech recognition using OpenAI's Whisper API and natural language processing using GPT-4-based models to transform raw meeting audio into structured, actionable outputs. A production-grade full-stack architecturecomprisingReactfor the frontend, FastAPI for the backend, and MongoDB for data storage ensures scalability andreliability.AutoScribeprovides users with accurate transcripts,concisesummaries,structured action item lists, and email-delivered reports, thereby enhancing productivity and decision-making. Experimental evaluation on 50 real-world meetingrecordingsdemonstrates a Word Error Rate (WER) of 6.3% for transcription, aROUGEL F1 score of 0.71 for summarization, and an actionextraction F1 score of 85.4%. These results confirm AutoScribe's effectiveness as a comprehensivemeetingintelligencesolution, particularly relevant in remoteandhybridworkenvironments

Key Words: AI, Automatic Speech Recognition, Meeting Assistant, Natural Language Processing, GPT-4, Whisper API, Summarization, Action Item Extraction, FastAPI, MongoDB

1.INTRODUCTION

In today's dynamic workplace, meetings serve as the cornerstoneoforganizationalcommunication,collaboration, and decision-making. Whether conducted in-person, remotely, or in hybrid formats, meetings facilitate knowledge exchange and strategic planning. However, a persistentchallengeremains thedocumentationofthese meetings. Studies indicate that professionals spend approximately 31 hours per month in unproductive meetings, with a significant portion of time wasted on follow-up and re-clarification due to poor or absent documentation[1].

Traditionalapproachestomeetingdocumentationinclude manual note-taking, which is inherently subjective and error-prone. Note-takers often miss critical points, misinterpret statements, or fail to capture action items accurately. This problem is amplified in fast-paced, multispeakermeetingswheresimultaneousconversationsoccur.

Theresultingdocumentationisoftenincomplete,inaccurate, or overly verbose, rendering it less useful for decisionmakingandfollow-up.

TheadventofArtificialIntelligence(AI),particularlyinthe domainsofAutomaticSpeechRecognition(ASR)andNatural Language Processing (NLP), has opened new avenues for automating meeting documentation. AI-powered systems can transcribe spoken language in real-time or from recordings,extractsemanticmeaning,andgenerateconcise summaries all with minimal human intervention. Such systemsnotonlyimproveaccuracybutalsoreducecognitive loadonmeetingparticipants,allowingthemtofocusonthe discussionratherthandocumentation.

AutoScribeisanAI-poweredmeetingassistantdesignedto address these challenges. The system accepts audio recordings of meetings, transcribes them using OpenAI's WhisperAPI,processesthetranscriptionsthroughGPT-4based NLP models to generate structured summaries and extract action items, and delivers these outputs to users throughanintuitivewebinterface.Thesystemalsosupports emaildeliveryofreportsandprovidescentralizedstorage through MongoDB, making it suitable for team-wide deployment.

This paper presents the design, architecture, implementation, and evaluation of AutoScribe. Section 2 reviews related work. Section 3 describes the system methodology and architecture. Section 4 discusses the systemmodulesindetail.Section5presentsexperimental results.Section6concludesthepaperwithfuturedirections.

2. LITERATURE REVIEW

Automatedmeetingtranscriptionandsummarizationhave attractedsignificantresearchattentionoverthepastdecade. Early systems relied on rule-based approaches and HMMbased ASR engines, which suffered from poor accuracy in noisy environments and with accented speech [2]. The introduction of deep learning-based ASR systems, particularlyrecurrentneuralnetworksandlatertransformer architectures,dramaticallyimprovedtranscriptionquality.

Whisper,developedbyOpenAI,representsthestateofthe art in open-vocabulary ASR. Trained on 680,000 hours of multilingualspeechdata,Whisperachievesnear-human-level word error rates across diverse acoustic conditions and accents[3].Itsrobustnessmakesitparticularlysuitablefor

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

meeting environments where audio quality can vary significantly.

In the domain of NLP-based summarization, extractive methods whichselectkeysentencesfromtheoriginaltext were dominant in early systems. However, abstractive summarization using transformer-based language models such as BERT, T5, and GPT variants has shown superior performance in generating coherent, contextually appropriate summaries [4]. GPT-4, with its instructionfollowingcapabilityandlargecontextwindow,isparticularly well-suited for meeting summarization and action item extractiontasks.

Several commercial meeting assistants exist, including Otter.ai, Fireflies.ai, and Microsoft Teams' built-in transcription.Whileeffective,thesetoolsaretypicallyclosedsource,subscription-based,andofferlimitedcustomization. AcademicsystemssuchasAMIMeetingCorpustoolsandICSI

Meeting Corpus analyzers offer research value but lack production-readyinterfaces[5].AutoScribebridgesthisgap by offering an open, extensible, full-stack solution with a customizableAIpipeline.

Action item extraction identifying tasks, assignees, and deadlines from meeting transcripts remains an understudied problem. Prior work by Purver et al. (2007) approacheditasaclassificationtaskusingshallowfeatures [6]. Modern LLM-based approaches, leveraging in-context learningandpromptengineering,offeramoreflexibleand accuratealternative,whichAutoScribeexploits.

3. SYSTEM ARCHITECTURE

AutoScribe follows a layered, modular architecture comprising a React-based frontend, a FastAPI-powered backend, and a MongoDB database. The system adopts RESTful APIdesignprinciplesforcommunicationbetween layers,ensuringloosecouplingandeasyscalability.Figure1 illustratesthehigh-levelsystemarchitecture.

3.1 Frontend Layer: The frontend is developed using React.js, a component-based JavaScript framework. It provides an intuitive user interface for audio file upload, progresstracking,andresultsdisplay.Theinterfaceincludes modulesfortranscriptviewing,summarydisplay,actionitem listing,andreportdownload.Authenticationishandledvia JWTtokens,ensuringsecuresessionmanagement.

3.2 Backend Layer: ThebackendisbuiltonFastAPI,ahighperformance Python web framework. FastAPI's asynchronous capabilities allow concurrent processing of multiplemeetinguploads.Thebackendexposesendpoints foruserauthentication,fileupload,transcriptiontriggering, summary generation, and data retrieval. All API calls are documented automatically via FastAPI's built-in OpenAPI (Swagger)interface.

3.3 AI Processing Pipeline: TheAIpipelineconstitutesthe coreofAutoScribe.Uponreceivinganaudiofile,thepipeline invokestheWhisperAPIforspeech-to-textconversion.The

resultingtranscriptisthenpassedtotheGPT-4APIthrough carefullyengineeredpromptsforsummarizationandaction extraction. The pipeline is designed to be modular individual components can be swapped or upgraded independentlyasbettermodelsbecomeavailable.

3.4 Database Layer: MongoDB,adocument-orientedNoSQL database,isusedfordatapersistence.Eachmeetingisstored as a document containing the raw transcript, generated summary, extracted action items, metadata (user ID, timestamp,duration),andemaildeliverystatus.MongoDB's flexibleschemaaccommodatesvaryingmeetingstructures withoutrequiringpredefinedtableschemas.

4 METHODOLOGY

AutoScribe'sprocessingpipelinefollowsasequential,fivestageworkflowasdescribedbelow.

Stage 1 Audio Upload: Usersuploadmeetingrecordings throughtheReactfrontend.SupportedformatsincludeMP3, MP4, WAV, and M4A. The frontend validates file type and size before transmitting the file to the backend via a multipartHTTPPOSTrequest.Thebackendstoresthefile temporarilyinlocalstorageforprocessing.

Stage 2 Transcription: Theaudiofileisforwardedtothe OpenAIWhisperAPI.Whisperperformsend-to-endspeech recognition, producing a raw text transcript with timestamps.Thesystemusesthe'whisper-1'model,which supportsover50languages.Speakerdiarization(attributing speechtoindividualspeakers)isappliedinpost-processing usingpyannote.audio,providingspeaker-labeledtranscripts wherepossible.

Stage 3 Summarization: The raw transcript is segmentedandpassedtotheGPT-4API.Acustomsystem prompt instructs the model to generate a structured summarycovering:(a)meetingagenda,(b)keydiscussion points,(c)decisionsmade, and(d)unresolvedissues.The summaryisformattedinmarkdownfor easyrendering in thefrontend.

Stage 4 Action Extraction: AseparateGPT-4callismade with a prompt engineered specifically to identify action items.ThemodelextractstasksinastructuredJSONformat containing: task description, assignee (if mentioned), deadline(ifmentioned),andprioritylevel.Thisstructured outputisdirectlyrenderedasaninteractivechecklistinthe frontend.

Stage 5 Storage and Delivery: Thetranscript,summary, and action items are stored in MongoDB under the user's account.Userscanoptionallytriggeranemaildeliveryofthe meeting report using SendGrid integration. Reports are formattedasHTMLemailswithinlineCSSforcompatibility acrossemailclients.

5. IMPLEMENTATION DETAILS

5.1 Technology Stack: Thecompletetechnologystackfor AutoScribe is summarized in Table 1. The choice of each

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

technologywasdrivenbyperformance,ecosystemmaturity, anddeveloperproductivityconsiderations.

Table 1: AutoScribe Technology Stack

Component Technology

Frontend

React.js,Axios,TailwindCSS

Backend FastAPI(Python3.10)

SpeechRecognition OpenAIWhisperAPI

NLP/Summarization OpenAIGPT-4API

SpeakerDiarization pyannote.audio

Database MongoDB(Atlas)

Authentication JWT(JSONWebTokens)

EmailDelivery SendGridAPI

Deployment Docker,Nginx,AWSEC2

5.2 API Integration: IntegrationwithOpenAIAPIsfollowsa secure,server-sidepattern.ThebackendstoresAPIkeysin environmentvariables,neverexposingthemtothefrontend. Requests are rate-limited using FastAPI middleware to complywithOpenAI'susagepolicies.Errorhandlingcovers APItimeout,tokenlimitexceeded,andinvalidaudioformat scenarios.

5.3 Security Implementation: AutoScribe implements OAuth2.0-compliantauthenticationusingJWTtokenswitha 24-hourexpiry.Passwordsarehashedusingbcryptwitha saltfactorof12.AllAPIendpointsareprotectedwithBearer token authentication except the registration and login endpoints.HTTPSisenforcedviaNginxreverseproxywith Let'sEncryptSSLcertificates.

5.4 Frontend Design: The React frontend employs a component-based architecture with React Router for navigation.Keycomponentsinclude:AudioUploader(dragand-drop file upload with progress indicator), TranscriptViewer (scrollable, searchable transcript with speaker labels), SummaryPanel (formatted meeting summary), ActionBoard (interactive task checklist), and HistoryDashboard(paginatedmeetinghistory).

6. EXPERIMENTAL EVALUATION

To evaluate AutoScribe's performance, a dataset of 50 meeting recordings was collected from consenting participants. Meetings ranged from 15 to 90 minutes, covering topics including project reviews, academic seminars, and team standups. Performance was assessed across four dimensions: transcription accuracy, summarization quality, action extraction precision, and systemresponsetime.

6.1 Transcription Accuracy: WordErrorRate(WER)was used as the primary metric for transcription quality. AutoScribeachievedanaverageWERof6.3%acrossthetest dataset,comparedto18.7%forGoogleSpeech-to-Text(free tier) and 9.1% for Azure Cognitive Services. Performance

was consistently better in low-noise environments, with WERaslowas3.1%forstudio-qualityrecordings.

Table 2: Transcription WER Comparison

6.2 Summarization Quality: Summarization quality was assessed using ROUGE-L scores against human-generated reference summaries. AutoScribe achieved a ROUGE-L F1 score of 0.71, indicating strong overlap with human summaries.Qualitativeevaluationbydomainexpertsrated 86% of generated summaries as 'accurate' or 'highly accurate'. Common failure modes included overcompression of technical discussions and occasional omissionofnuancedcontext.

6.3 Action Extraction Precision: Foractionitemextraction, precision and recall were measured against manually annotatedgroundtruth.AutoScribeachievedaprecisionof 88.4%andrecallof82.7%,resultinginanF1scoreof85.4%. Themodelperformedbestwhenactionitemswereexplicitly stated(e.g.,'JohnwillsubmitthereportbyFriday')andless accuratelyonimplicitcommitments.

6.4 System Performance: Averageend-to-endprocessing timefora30-minutemeetingrecordingwas4.2minuteson thedeployedserver(AWSEC2t3.medium).Thisincludes2.8 minutes for Whisper transcription, 0.7 minutes for summarization, 0.4 minutes for action extraction, and 0.3 minutesforstorageoperations.Forreal-timeapplications, the system can process audio in streaming chunks with approximately1.3xreal-timespeed.

7. USE CASE SCENARIOS

7.1 Corporate Team Meetings: AutoScribeisparticularly valuable for weekly team standups and project review meetings. Integration with calendar systems (Google Calendar,Outlook)enablesautomaticmeetingdetectionand scheduling of post-processing. Action items are synchronized with project management tools such as Jira andTrelloviawebhookintegrations.

7.2 Academic and Research Seminars: Research group meetings and conference presentations benefit from AutoScribe's ability to handle technical vocabulary. The system's multi-language support (50+ languages) accommodates international academic collaborations. Lecturerecordingscanbeprocessedtogeneratestudynotes andhighlightkeyconcepts.

International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056

Volume: 13 Issue: 04 | Apr 2026 www.irjet.net p-ISSN: 2395-0072

7.3 Remote and Hybrid Work: Inthepost-pandemiceraof distributedteams,AutoScribeaddressesthedocumentation gap in virtual meetings. Unlike native conferencing tools, AutoScribeisplatform-agnostic itworkswithrecordings from Zoom, Microsoft Teams, Google Meet, or any audio source.Thisflexibilitymakesituniversallyapplicableacross organizationalITlandscapes

8. LIMITATIONS AND FUTURE WORK

Despite its strong performance, AutoScribe has several limitations.First,speakerdiarizationaccuracydegradesin meetings with more than six participants due to voice similarity. Future work includes fine-tuning speaker embedding models on meeting-domain data. Second, the systemcurrentlydoesnotsupportreal-timetranscription only post-meeting processing. A streaming mode using Whisper'sreal-timeAPI,currentlyinbeta,isplannedfora futurerelease.

Third,GPT-4-basedsummarizationintroduceslatencyand costpermeeting.Exploringsmaller,fine-tunedmodels(e.g., Mistral-7BorLLaMA-3fine-tunedonmeetingdata)ascosteffectivealternativesisapriorityforfutureresearch.Fourth, the current action extraction module lacks the ability to tracktaskcompletionstatusovertime.Futureiterationswill include a project-level dashboard with task tracking, deadlinereminders,andcompletionanalytics. Additionally, privacy compliance in enterprise settings requires on-premise deployment options. Future releases will support local model inference using quantized LLMs, enabling sensitive-data processing without cloud API dependencies.Integrationwithenterpriseidentityproviders (LDAP,SAML2.0)isalsoplanned

9. CONCLUSIONS

This paper has presented AutoScribe, an AI-powered meeting assistant that automates transcription, summarization, and action item extraction from meeting recordings.ByintegratingOpenAI'sWhisperASRandGPT-4 NLP models within a production-grade full-stack architecture, AutoScribe delivers accurate, structured meetingdocumentationwithminimalusereffort. Experimental evaluation on 50 meeting recordings demonstratesthatAutoScribeachievesa WERof6.3% for transcription,aROUGE-LF1of0.71forsummarization,and anactionextractionF1of85.4%.Theseresultsvalidatethe effectiveness of combining state-of-the-art ASR and LLM technologiesformeetingintelligenceapplications. AutoScribe is particularly relevant in today's hybrid work environments,whereefficientdocumentationandfollow-up arecriticalforteamproductivity.Byeliminatingtheburden of manual note-taking, AutoScribe enables participants to engagemoredeeplyindiscussionswhileensuringthatno criticalinformationislost.Futureworkwillfocusonrealtime processing, on-premise deployment, and integration withenterpriseproductivityecosystems.

REFERENCES

1. J. M. Perlow, C. Hadley, and E. Eun, "Stop the MeetingMadness,"HarvardBusinessReview,Jul.Aug.2017.

2. S. Young et al., "The HTK Book (for HTK Version 3.4)," Cambridge University Engineering Department,2006.

3. Radford,J.W.Kim,T.Xu,G.Brockman,C.McLeavey, and I. Sutskever, "Robust Speech Recognition via Large-Scale Weak Supervision," arXiv preprint arXiv:2212.04356,2022.

4. T. Brown et al., "Language Models are Few-Shot Learners," in Advances in Neural Information ProcessingSystems(NeurIPS),vol.33,2020.

5. J.Carletta etal.,"TheAMIMeeting Corpus:A Preannouncement," in Machine Learning for MultimodalInteraction,Springer,2006.

6. M.Purver,J.Dowding,J.Niekrasz,P.Ehlen,andS. Peters,"DetectingandSummarizingActionItemsin Multi-Party Dialogue," in Proc. 8th SIGdial Workshop on Discourse and Dialogue, Antwerp, Belgium,2007.

7. OpenAI, "OpenAI API Documentation," [Online]. Available: https://platform.openai.com/docs. [Accessed:Mar.2025].

8. FastAPI, "FastAPI Documentation," [Online]. Available:https://fastapi.tiangolo.com. [Accessed: Mar.2025].

9. ] MongoDB, Inc., "MongoDB Documentation," [Online]. Available: https://www.mongodb.com/docs.[Accessed:Mar. 2025].

10. React,"ReactDocumentation,"[Online].Available: https://reactjs.org.[Accessed:Mar.2025].