Paper hyperlink: MLoRA: Multi-Domain Low-Rank Adaptive Network for Click-Through Rate Prediction
That is an trade paper from Alibaba. They introduce personalized Lora element to multi-domain CTR mannequin, and located it’s useful in 2 factors.
1. Mitigating area collapse attributable to information sparsity and disparate information distributions in order that to enhance mannequin efficiency
2. Scaling up with much less mannequin parameters
Mannequin Structure
They talked about utilizing consumer, merchandise and context options. From their figures, the mannequin could share options throughout multi domains.
Within the mannequin structure, they break up every layer of the mannequin into a standard half and a personalised half. Apply MLoRA to the personalised half.
Contemplating the distinction amongst CTR mannequin layers, they utilized a scaling issue to the r in LoRA
Based mostly on their figures, MLoRA element is utilized to every interplay MLP layer after the embedding layer.
Experiment Setup
They evaluated the strategy on Taobao multi-domain and Amazon multi-domain and Movielens-gen datasets. And noticed MLoRa tailored higher efficiency on every area.
My 3 Takeaways:
1. The ideology of beginning considering from personalization illustration within the forluma, then adapting LoRA to it.
2. The interpretation of their personalized r, why fastened r as authentic LoRA doesn’t work effectively in CTR fashions. Their cause behind is that CTR fashions normally exhibit important variations in community width between layers not like NLP duties
3. Private ideas, they use LoRA on each layer, however we are able to additionally attempt a special strategy.
#Machine Studying #RecommendationSystem #CTR #LoRA # LLM Software #Alibaba #2024 #E-Commerce #Procuring
