Multimodality

[rmuhawieh]

Hello, I'm thinking of ways to improve my privateGPT and hoping to be able to work with images and potentially audio as a form of context in addition to the already working textual documents. I'm essentially looking for multi-modality. I'm currently using mistral but wondering what it would take to make privateGPT multimodal, as I've heard of llms like LLAVA, CLIP, etc. but not sure what the implementation would look like to make this possible. Thanks