Novel differentially non-public mannequin utilizing gradient flows outlined on an optimum transport metric.
This Analysis Card introduces a novel, theoretically grounded technique for differentially non-public generative modeling by leveraging a easy mathematical course of, reaching high-fidelity knowledge era with robust privateness ensures and decrease computational prices in comparison with conventional approaches.
- Title: Differentially Private Gradient Flow based on the Sliced Wasserstein Distance
- Quick Title: Novel differentially non-public mannequin utilizing gradient flows outlined on an optimum transport metric.
- Authors: Ilana SEBAG, Muni Sreenivas Pydi, Jean-Yves Franceschi, Alain Rakotomamonjy, Mike Gartrell, Jamal Atif, Alexandre Allauzen
- Workforce : RSC.FDL. Collaboration with : Miles Workforce, LAMSADE, Université Paris-Dauphine, PSL College, CNRS and ESPCI PSL.
- Standing : Revealed at TMLR (01/2025)
- Class : Privateness, Generative Modeling, gradient flows.
Safeguarding knowledge has turn into important on this period of widespread AI adoption. Generative modeling, particularly, poses distinctive challenges as a result of its skill to be taught and replicate intricate knowledge distributions, which dangers exposing delicate data from the unique dataset if the mannequin is educated with none privateness part. Whereas current approaches like including noise to the gradient (DP-SGD) or utilizing differentially non-public losses for generator-based strategies are efficient, they face limitations in balancing three key features:
- Privateness (How effectively is the info protected against privateness assaults?),
- Constancy (How real looking and high-quality is the info generated by the mannequin?),
- Computational effectivity (How a lot computation and assets are required to coach the mannequin?).
By introducing a novel differentially non-public algorithm primarily based on gradient flows and the Gaussian-smoothed Sliced Wasserstein Distance, we purpose to decrease knowledge leakage whereas reaching high-fidelity knowledge era beneath low privateness budgets and lowered computational prices. This principled different addresses unexplored areas in privacy-preserving generative modeling, advancing the sphere towards extra accountable AI growth.
On this work, we current a novel theoretical framework for a differentially non-public gradient circulate of the sliced Wasserstein distance (SWD), which has not been explored beforehand as a technique to assure differential privateness for generative AI fashions. Our method includes defining the gradient circulate on the smoothed SWD. Though the Gaussian smoothing technique seems easy, it introduces important theoretical challenges, notably relating to the existence and regularity of the gradient circulate resolution.
To handle these complexities, we set up the “continuity equation” for our new gradient circulate of the smoothed SWD, leading to a smoothed velocity discipline that governs how the generated knowledge is privately produced. This permits us to discretize the continuity equation from the earlier step, right into a Stochastic Differential Equation (SDE) that ensures the gradient circulate maintains differential privateness. Notably, we present that after discretization, the smoothing course of within the drift time period capabilities as a Gaussian mechanism, guaranteeing that the privateness finances is rigorously tracked all through the method.
On the theoretical entrance, our contribution is important as we show, for the primary time within the literature, the existence and regularity of the gradient circulate for the Gaussian-smoothed Sliced Wasserstein distance (GSW). The proof methods we use, impressed by earlier works, require intensive modification to deal with the distinctive traits of the GSW. This novel theoretical end result lays the muse for future work on differential privateness gradient flows, opening the door to new prospects and enhancements in privateness preserving AI.
From an experimental standpoint, we present that our proposed method outperforms the baseline DPSWgen mannequin, which makes use of a generator-based structure with the differentially non-public Sliced Wasserstein loss, throughout varied privateness budgets (ranges of privateness). Our technique not solely achieves higher FID scores but in addition generates higher-quality samples, demonstrating the sensible viability and superior efficiency of our method in safeguarding privateness whereas producing high-fidelity generative fashions.
