georg-jung / FastBertTokenizer Star 42 Code Issues Pull requests Fast and memory-efficient library for WordPiece tokenization as it is used by BERT. nlp machine-learning natural-language-processing ai tokens nlp-machine-learning bert tokenization wordpiece bert-embeddings wordpiece-tokenization llm Updated Jan 30, 2025 C#
theQuert / inlpfun Star 3 Code Issues Pull requests NLP Code Snippets and Conference related nlp machine-learning algorithms transformer colab papers mask bert reformer bert-models papers-with-code cloud-tpu wordpiece-tokenization Updated Dec 5, 2023 Jupyter Notebook
SeanLee97 / BertWordPieceTokenizer.jl Star 3 Code Issues Pull requests WordPiece Tokenizer for BERT models. nlp bert wordpiece wordpiece-tokenization transfomers Updated Mar 8, 2022 Julia
SpydazWebAI-NLP / SpydazWebAI_NLP_Models Star 1 Code Issues Pull requests Word/Image/Audio Embedding models, Tokenizer models, Ngram language models, MatrixModels, Corpus building, Vocabulary Building, Language modelling word2vec matrix tokenizer embeddings bayesian-inference cooccurrence latent-dirichlet-allocation vocabulary-builder mutual-information tokenization minhash-lsh-algorithm ngram-language-model image2vec bpe wordgrams wordpiece-tokenization wordpeice audio2vec word2word-matrix Updated Aug 21, 2023 Visual Basic .NET