I wont be taking a deep learning course during my Junior year at Bridgewater State University due to low enrollment. I instead took another computer science elective that lets me work on any project idea I would like and I would have another Undergraduate Research Project under my belt.
For this class, I wanted to pick a project idea that would strengthen my knowledge of all the stages of machine learning.
One of the first and boldest ideas to come to mind was to build a Speech-To-Text model in python and train it to quickly transcribe a stream of audio data.
I couldnt shake this idea because many tech companies offer this as a service in the form of API or a product and its probably %50 of my own JARVIS.
I really dont care if its off of my compute and as a bonus to saving money tokens-wise, it would mean my audio data wouldnt go through third parties.
Additionally, Ive been thinking alot about user interfaces, and I cant think of a better user interface than speech if its done right.
Transcription: The user speaks english and it processes the stream of audio and outputs a stream of tokens.
Accuraccy: The model produces the correct token for each chunk processed, even when nothing is said.
Speed: The model's output should be no longer .
First, I need to know what language, frameworks and tools I will use to make this project
Language: Python 3.14
Python 3.14 Packages:
librosa: I will use this libraries tools to produce log_mel_spectrograms of audio chunks. These represent audio data's frequency and amplitude over time aswell as tuned to human's perception of the audio-data.
matplotlib: This library will be used for viewing chunks of data to test for model and algorithm accuracy during development phase.
pytorch: I will use this library to design the model architecture and to automatically train the model.
noisereduce: This library will reduce noise not needed for processing the speech data.
NVIDIA: NeMo token duration tools: I will use these tools to produce token durations of audio data to train my model's Token duration calculation.
Next im going to need to know where to get my data.
There are plenty of open-source datasets that are precisely what I am looking for, this will be used for general training on many token patterns and durations
Moreover, I would definately want it fine-tuned to my speech patterns after initial training with open-source data sets.
INPUT: Overlapping 35ms segments of an audio signal that is pre-processed and is represented as a log mel spectrogram
OUTPUTS: BPE Seintence Piece token (1025 possible outputs), token duration (optional)
Architecture: RNN Transducer & Token Duration Preduction
The RNN Transducer consists of an encoder to process input log mel spectrograms and a decoder to process corresponding tokens
I can decide on my data later, right now I want to dig into the meat and beans of deep learning model architecture.
I wrote the last line and I dont think I can explain my chosen architecture well right now but its going to be a psuedo TDT architecture similar to how NVIDIA's NeMo TDT model works. The gist of it is that it has an output for token duration aswell as the token present in each log mel spectrogram over time.
I will have to find some tricks and fill the gaps in my knowledge to write more about it so I am going to start with some facts.
I have attached a link here showing my progress, but all I have done is get log mel spectrograms from each of the files and some extra functionality I might need later.
The data will have to be in the format:
File(str): file that the data is from
Duration(float): file duration in seconds
Text(str): what was spoken in the file
Tokens(int[]): BPE Sientence Piece token encoding of the text
Token Durations(int[]): length in seconds of each token above
Whats next is that I need to use a tool or make an algorithm to generate the Tokens and Token Durations.