How I Won the “Mostly AI” Synthetic Data Challenge

I within the Mostly AI Prize and gained each the FLAT and SEQUENTIAL information challenges. The competitors was a unbelievable studying expertise, and on this submit, I need to present some insights into my profitable resolution.

The Competitors

The aim of the competitors was to generate an artificial dataset with the identical statistical properties as a supply dataset, with out copying the info.

The competitors was cut up into two unbiased challenges:

1. FLAT Information Problem: Generate 100,000 data with 80 columns.
2. SEQUENTIAL Information Problem: Generate 20,000 sequences (teams) of data.

To measure the standard of the artificial information, the competitors used an General Accuracy metric. This rating measures the similarity between the artificial and supply distributions for single columns (univariates), pairs of columns (bivariates), and triples of columns (trivariates) utilizing the L1 distance. Moreover, privateness metrics like DCR (Distance to Closest Document) and NNDR (Nearest Neighbor Distance Ratio) have been used to make sure submissions weren’t simply overfitting or copying the coaching information.

Answer Design

Initially, my aim was to create an ensemble of a number of totally different state-of-the-art fashions and mix their generated information. I experimented quite a bit with totally different fashions, however the outcomes didn’t enhance as a lot as I had hoped.

I pivoted my strategy and targeted on post-processing. First, I skilled a single generative mannequin from the Mostly AI SDK, and as a substitute of producing the required variety of samples for the submission, I oversampled to create a big pool of candidate samples. From this pool, I then chosen the ultimate output in a approach that matches the statistical properties of the supply dataset far more carefully.

This strategy led to a considerable leap within the leaderboard rating. For the FLAT information problem, the uncooked artificial information from the mannequin scored round 0.96, however after post-processing, the rating jumped to 0.992. I used a modified model of this strategy for the SEQUENTIAL information problem, which yielded the same enchancment.

My ultimate pipeline for the FLAT problem consisted of three predominant steps:

1. Iterative Proportional Becoming (IPF) to pick out an outsized, high-quality subset.
2. Grasping Trimming to scale back the subset to the goal measurement by eradicating the worst-fitting samples.
3. Iterative Refinement to shine the ultimate dataset by swapping samples for higher becoming ones.

Step 1: Iterative Proportional Becoming (IPF)

Step one in my post-processing pipeline was to get a powerful preliminary subset from the oversampled pool (2.5 million generated rows). For this, I used Iterative Proportional Fitting (IPF).

IPF is a classical statistical algorithm used to regulate a pattern distribution to match a recognized set of marginals. On this case, I needed the artificial information’s bivariate (2-column) distributions to match these of the unique information. I additionally examined uni- and trivariate distributions, however I discovered that specializing in the bivariate relationships yielded the very best efficiency whereas being computationally quick.

Right here’s the way it labored:

1. I recognized the 5,000 most correlated column pairs within the coaching information utilizing mutual data. These are a very powerful relationships to protect.
2. IPF then calculated fractional weights for every of the two.5 million artificial rows. The weights have been adjusted iteratively in order that the weighted sums of the bivariate distributions within the artificial pool matched the goal distributions from the coaching information.
3. Lastly, I used an expectation-rounding strategy to transform these fractional weights into an integer rely of what number of occasions every row must be chosen. This resulted in an outsized subset of 125,000 rows (1.25x the required measurement) that already had very robust bivariate accuracy.

The IPF step offered a high-quality place to begin for the following part.

Step 2: Trimming

Producing an outsized subset of 125,000 rows from IPF was a deliberate selection that enabled this extra trimming step to take away samples that didn’t match properly.

I used a grasping strategy that iteratively calculates the “error contribution” of every row within the present subset. The rows that contribute probably the most to the statistical distance from the goal distribution are recognized and eliminated. This course of repeats till solely 100,000 rows stay, guaranteeing that the worst 25,000 rows are discarded.

Step 3: Refinement (Swapping)

The ultimate step was an iterative refinement course of to swap rows from the subset with higher rows from the a lot bigger, unused information pool (the remaining 2.4 million rows).

In every iteration, the algorithm:

1. Identifies the worst rows inside the present 100k subset (these contributing most to the L1 error).
2. Searches for the very best substitute candidates from the surface pool that would cut back the L1 error if swapped in.
3. Performs the swap if it ends in a greater total rating.

Because the accuracy of the artificial pattern is already fairly excessive, the extra achieve from this course of is fairly small.

Adapting for the Sequential Problem

The SEQUENTIAL problem required the same strategy, however with two modifications. First, a pattern consists of a number of rows, linked by the group ID. Secondly, the competitors metric provides a measure for coherence. This implies not solely do the statistical distributions have to match, however the sequences of occasions additionally must be just like the supply dataset.

My post-processing pipeline was tailored to deal with teams and likewise optimize for coherence:

1. Coherence-Based mostly Pre-selection: Earlier than optimizing for statistical accuracy, I ran a specialised refinement step. This algorithm iteratively swapped complete teams (sequences) to particularly match the coherence metrics of the unique information, such because the distribution of “distinctive classes per sequence” and “sequences per class”. This ensured that we continued the post-processing with a sound sequential construction.
2. Refinement (Swapping): The 20,000 teams chosen for coherence then went by means of the identical statistical refinement course of because the flat information. The algorithm swapped complete teams with higher ones from the pool to attenuate the L1 error of the uni-, bi-, and trivariate distributions. A secret ingredient was to incorporate the “Sequence Size” as a function, so the group lengths are additionally thought-about within the swapping.

This two-stage strategy ensured the ultimate dataset was robust in each statistical accuracy and sequential coherence. Curiously, the IPF-based strategy that labored so properly for the flat information was much less efficient for the sequential problem. Subsequently, I eliminated it to focus computing time on the coherence and swapping algorithms, which yielded higher outcomes.

Making It Quick: Key Optimizations

The post-processing technique by itself was computationally costly, and making it run inside the competitors time restrict was a problem in itself. To succeed, I relied on a couple of key optimizations.

First, I lowered the info sorts wherever doable to deal with the large pattern information pool with out operating out of reminiscence. Altering the numerical sort of a giant matrix from 64-bit to 32 or 16-bit enormously reduces the reminiscence footprint.

Secondly, when altering the info sort was not sufficient, I used sparse matrices from SciPy. This system allowed me to retailer the statistical contributions of every pattern in an extremely memory-efficient approach.

Lastly, the core refinement loop concerned a variety of specialised calculations, a few of which have been very sluggish with numpy. To beat this, I used numba. By extracting the bottlenecks in my code into specialised capabilities with the @numba.njit decorator, Numba robotically translated them into extremely optimized machine code that runs at speeds similar to C.

Right here is an instance of how I wanted to hurry up the summation of rows in sparse matrices, which was a serious bottleneck within the authentic NumPy model.

import numpy as np
import numba

# This will make the logic run lots of of occasions quicker.
@numba.njit
def _rows_sum_csr_int32(information, indices, indptr, rows, Ok):
    """
    Sum CSR rows right into a dense 1-D vector with out creating
    intermediate scipy / numpy objects.
    """
    out = np.zeros(Ok, dtype=np.int32)
    for r in rows:
        begin = indptr[r]
        finish = indptr[r + 1]
        for p in vary(begin, finish):
            out[indices[p]] += information[p]
    return out

Nevertheless, Numba is just not a silver bullet; it’s useful for numerical, loop-heavy code, however for many calculations, it’s quicker and simpler to stay to vectorized NumPy operations. I counsel you to solely attempt it when a NumPy strategy doesn’t attain the required velocity.

Ultimate Ideas

Although ML fashions are getting more and more stronger, I believe that for many issues that Information Scientists are attempting to resolve, the key ingredient is commonly not within the mannequin. After all, a powerful mannequin is an integral a part of an answer, however the pre- and postprocessing are equally essential. For these challenges, a post-processing pipeline focused particularly for the analysis metric led me to the profitable resolution, with none extra ML.

I discovered quite a bit on this problem, and I need to thank Mostly AI and the jury for his or her nice job in organizing this unbelievable competitors.

My code and options for each challenges are open-source and could be discovered right here: