Purpose: Create a GenAI assistant that may perceive driver intent by way of textual content and interpret visible context by way of dashboard icons or alerts — and reply naturally, like a educated human passenger.
The system processes two key enter streams in parallel:
- Textual content: Pure language queries from the motive force
Instance: “What does this pink triangle imply?” - Imaginative and prescient: A dashboard picture or icon captured by way of an inside digicam or system logs
Instance: Cropped warning mild or dwell dashboard view
These inputs undergo separate encoding pathways:
- Textual content Encoder: GPT, BERT, or T5
- Imaginative and prescient Encoder: CLIP, ViT, or ResNet
Each are then fused utilizing cross-attention mechanisms (e.g., BLIP-2, Flamingo), forming a joint embedding handed right into a multimodal decoder — usually a fine-tuned LLaMA or GPT variant skilled on automotive language.
The mannequin then generates a pure language response, similar to:
“That’s the tire strain warning. I like to recommend checking all 4 tires. Would you want me to discover a close by service station?”
