AI Seminar- Nexa AI – Functional Tokens for On-device Multimodal Models

Speaker: Alex Chen, CEO + Founder of Nexa AI and Zack Li, CTO + Co-Founder of Nexa AI, Nexa AI

Talk Title: Nexa AI -“ Functional Tokens for On-device Multimodal Models

Abstract: Zoom meeting ID: 944 0958 4905Passcode: 822247 Tokenizing corpora into semantic tokens has proven effective for large language models. However, this approach encounters challenges when applied to function calls, leading to inaccuracies and hallucinations. To address this issue, we have pioneered a new training methodology using functional tokens, transforming complex function calling tasks into language completion tasks. We also released Octopus-series models using functional tokens and achieved GPT4 level function calling accuracy with 2B parameter size. Our Octopus-V2 model achieved 35 times faster inference speed up and 70 times more energy efficiency compared to the RAG plus Llama3 solution, and is four times faster than OpenAI’s GPT-4O. The functional token is then applied to Octopus-V3, a sub-billion multimodal model, adept at both text and images, and fluent in English and Mandarin. Furthermore, Octopus-V4 extends these capabilities into a graph network structure, with Octopus-V2 as the master node and integration with other open-source models as worker nodes, Octopus-V4 achieved 74.8 MMLU and outperforms GPT3.5, and applied for cloud and edge collaboration. Nexa’s Octopus-V2 models ranked 2nd place among half a million models on HuggingFace between Apr 2 and Apr 15, surpassing XAI grok and Databrick DBRX model during that period, and was mentioned by Google Gemma team during the 2024 Google IO. Nexa’s Octopus models have also attracted industrial collaboration interest from AWS, Google, Volkswagen US, Qualcomm, ByteDance, Stellantis, Zoom, and more.

Biography: Alex Chen is the CEO and founder of Nexa AI, with PhD in Mechanics and Computation from Stanford University. His research interests lie in AI agent development empowered by large language models. He is a serial entrepreneur and served as President of the Chinese Entrepreneur Organization before. He is also a gold medalist in the Mathematics Olympiad. Zack Li is the CTO and co-founder of Nexa AI. Before this, he accumulated four years of industrial experience in on-device AI at Google and Amazon Lab126, focusing on model deployment, performance optimization, and edge-cloud collaboration. He received an MS in Operation Research from Stanford University. Alex and Zack are founders of Nexa AI and have authored Octopus series models. Nexa AI builds lightweight but powerful multimodal models for AI agents and provides on-device SDK infra to make models run fast and energy-efficiently. For more information, visit https://www.nexa4ai.com/ If speaker approves to be recorded for this AI Seminar talk, it will be posted on our USC/ISI YouTube page within 1-2 business days: https://www.youtube.com/user/USCISI. Subscribe here to learn more about upcoming seminars: https://www.isi.edu/events/

Host: Abel Salinas and Justina Gilleland

More Info: https://www.isi.edu/events/5009/nexa-ai-functional-tokens-for-on-device-multimodal-models/

Webcast: https://www.youtube.com/watch?v=MxAiFHSRrwQ

Location: Information Science Institute (ISI) - Virtual Only

WebCast Link: https://www.youtube.com/watch?v=MxAiFHSRrwQ

Audiences: Everyone Is Invited

Contact: Pete Zamar

Event Link: https://www.isi.edu/events/5009/nexa-ai-functional-tokens-for-on-device-multimodal-models/