API That Went Through Five Revisions

I was building an analysis engine for a serverless function that loads a hierarchical metric tree from a data warehouse (think: a top-level KPI decomposes into sub-metrics algebraically, each decomposing further across ~100 nodes), and propagates what-if changes through the tree. The agent wrote the first version in hours. It worked. Tests passed.

What followed was five rounds of refactoring the same module. Each iteration, the agent rewrote the code, updated tests, redeployed. I kept pushing: “no, these columns aren’t tree nodes, they’re raw inputs to derived ratios. The naming should reflect that.” A flat lookup dict became a typed configuration with a dataclass. Each round got cleaner.

I was doing design work, not typing work. The coding agent handled mechanical refactoring: renaming across files, updating assertions, regenerating query logic. I decided whether the abstraction made sense.

The code review came back with a pointed question: “Do we still need this data-loading mode?” I initially argued yes! The reviewer was right though: another tool already handled data inspection. And because each service should do one thing, we ripped the duplicated data loading module out, rewrote the interface from four modes to two, updated three packages, revalidated all nodes, and merged. The agent did all the mechanical work and I was responsible for detecting “smelly” work and push back when necessary.

Writing Math Proofs

This is where it got interesting. My paper extends a recent sensitivity analysis framework from linear models to double machine learning. The theory section was a 23-line stub. My collaborator said it needed to become the centerpiece. I had the agent download the original paper from arxiv, extract the proofs, and explain each theorem. Key finding was that the original proofs use linear algebra projections everywhere and they don’t extend to nonparametric models. We couldn’t just cite them. We needed original proofs.

After a comprhensive literature review, a three-tier proof structure emerged in one session. Proposition 1: the linear case reduces exactly to the original framework. Proposition 2: additive models, consistency via decomposition. Proposition 3: general nonlinear case via concentration inequalities. The agent wrote the LaTeX. I checked the math. Then I told it to run an adversarial audit on its own proofs.

It found two critical errors. First, it used an inequality that requires independent random variables. Our sampling scheme produces negatively correlated variables. Wrong inequality. We switched to one that handles dependent sampling. Second, it claimed a variance function is always concave. I was skeptical. It produced a counterexample with two interacting binary variables where the marginal information gain *increases* which is the opposite of concavity. The claim was false.

The agent wrote a proof, disproved part of its own proof with a concrete counterexample, then rewrote the proposition with an honest three-case treatment. I’ve worked with human collaborators who wouldn’t catch that. At this point it feels like if I throw as many tokens as time allows and provide clean and organized context to the agent, solving great number of problems is just a matter of time (not gonna argue that its able to find the pareto optimal solution or not).

Thousands of Cloud Training Jobs

Next, for the above mentioned paper we needed a massive simulation study with nearly 3,000 configurations across effect sizes, confounding strengths, and increasing sample sizes. Each runs as a cloud training job. The agent wrote the dispatch script.

The dispatch job died at job 194. Root cause: O(n) bottleneck. For every new job, the script polled the status of all previously launched jobs. API rate limits turned this into a 40-second check per iteration. Throughput collapsed. Credentials expired.

But to fix this, the agent researched the cloud API’s behavior, found that job creation fails synchronously at the quota limit rather than queuing, and redesigned as fire-and-forget with retry-on-quota. The new version dispatched at maximum API rate, handled quota limits with backoff, and added resume flags for credential expiry. After this fix, all jobs running.

Then it collected results, identified which hyperparameter actually dominates model bias (not the one I expected a 3x range from a single parameter), and generated the appendix analysis. Dispatch, collection, analysis in ONE day. That’s a week/s of deep focus work without the coding agent.

Where It Goes Wrong

One time the agent submitted a code review for three packages without deploying or running end-to-end validation. I managed to catch it and told the agent: “We must test the entire pipeline before sending the review. Actually make sure this never happens again.” To handle this, we added a pre-submission validation checklist as a permanent rule. The learning from this is that the agents at the current state always try to optimize for task completion and will skip validation unless we explicitly require it. One solution that seems to work well for this is to set up “hooks” that run automatically after builds. This way you can make the validation deteministic and dont rely on the agent to remember the routine, which becomes more of an issue with the increase in context sizes.

Another pattern that I need to mention was when the paper draft had stale numbers from an old experiment. Agent’s adversarial audit caught it, which led to a key statistic changing meaningfully when we switched to proper cross-validation. The old numbers were wrong and had been sitting in the draft for days. The agent can catch errors, but only when you tell it to look. It doesn’t spontaneously doubt its own previous output, and in most cases, it will say that everything has been implemented and tested thoroughly. To create the adversarial audit, I have worked with the agent to read and compile the literature about what kind of steering intructions and methods tend to work well with coding agents. As a result, we created a manually invoked instruction document that I invoke whenever something smells fishy. Works pretty well so far.

Delegation as the Core Skill

Here’s what I think most people miss. Using these tools well is not about prompt engineering or knowing the right magic words. It’s about delegation and evaluation which are the same skills you need to manage people, applied to a machine.

I write instruction sets loaded by task phase: thinking, researching, implementing, reviewing. I run adversarial audits where the agent attacks its own work. I enforce checklists. I push back when the abstraction is wrong. It’s less like pair programming and more like being a tech lead for a very fast, very literal engineer who occasionally produces brilliant work and occasionally tries to ship without testing.

The evaluation part is critical and underappreciated. When the agent writes a proof, I need to know enough math to verify it. When it refactors a data mapping, I need to understand the schema well enough to judge whether the new abstraction is better. When it dispatches cloud jobs, I need to understand the API limits to know if the retry logic makes sense. I think I now understand what people meant when they said “AI amplifies your existing expertise”. If you don’t have the expertise, it amplifies your ability to produce confident-looking garbage.

I wrote proofs, shipped a production API, dispatched thousands of training jobs, and debugged infrastructure WITHIN the same week and while training for a triathlon. A few months ago, that would have been a month of just the engineering work. The tools are crude, context windows degrade, and the agent drifts without anchoring. But for the kind of work where the hard part is the science and the design, and not the typing, it’s already a different game for me. And to think I’ve only started investing into learning agentic coding this January…..

Cheers, see yall next time!

1. The XML “attention fence”

The reason XML reads like a native interface to Claude comes down to how its self-attention heads were trained.

During pre-training, untrusted data gets wrapped in tags. That teaches the model to lower the weight it places on tokens inside something like <user_query> when it is predicting the next token for a system instruction. The tags end up acting as a structural fence, which is what keeps the model from mistaking your background data for a new command (sometimes called attention contamination).

2. Phase 1: supervised fine-tuning as behavioral cloning

This is the stage where the model learns the constitution, a rulebook of principles. It isn’t reasoning yet, it is mimicking.

The loop: the model generates a response, critiques it against the constitution, revises it, and then gets trained to clone that revised version. The mechanism is cross-entropy loss. For every token, the model outputs a probability vector over its full vocabulary (on the order of 100k tokens). You compare against a one-hot ground-truth vector , which is all zeros except a at the correct token’s index:

If the model put 40% on “Hello” when the truth was <thinking> at 100%, backpropagation adjusts weights across the network to make <thinking> the statistically favored choice next time.

3. Phase 2: RLAIF and PPO, the optimization stage

Once the model knows how to speak, this phase teaches it judgment, using reinforcement learning from AI feedback. The PPO objective looks like:

Three pieces worth pulling apart:

• The reward. An AI judge model scores responses. Follow the constitution and use structure correctly, and the response earns a positive scalar; the logic that produced it gets reinforced.
• The KL penalty. This is the part that matters. If the model tries to game the reward by drifting into gibberish the judge happens to like, the KL divergence term spikes and pulls it back toward the stable language model from Phase 1.
• PPO clipping. Proximal Policy Optimization caps how far the weights can move per update, which prevents model collapse from a few outlier rewards.

4. Scaling: the infrastructure bill

Classic RLHF is bottlenecked by human reading speed. RLAIF removes the human from the inner loop, so the feedback runs at GPU-cluster speed instead. The cost isn’t labor at that point, it is the electricity and compute to run these self-improvement loops at scale.

The takeaway. Using XML tags isn’t about being tidy. It is about lining your prompt up with the statistical patterns the model’s weights were optimized to reward in the first place.

What OVB actually is

Omitted variable bias shows up when you leave out a variable that is correlated with both your treatment of interest and your outcome. The model has nowhere to put that variable’s influence, so it smears it onto the coefficients you did include, and your estimate of the causal effect drifts up or down.

The classic example: regressing salary on years of education while leaving out ability. If ability raises both how much schooling someone gets and how much they earn, the education coefficient absorbs part of ability’s effect, and you overstate the return to education.

Two conditions have to hold for the bias to exist:

1. the omitted variable genuinely affects the outcome, and
2. it is correlated with an included regressor.

If either fails, omission is harmless. The direction of the bias follows the signs: same-sign correlations push the estimate up, opposite signs push it down. Reasoning through those signs without knowing the magnitude is what people call “signing the bias.”

The algebra

Suppose the true model is

but you omit and fit

Then the estimated coefficient on has expectation

where is the coefficient from regressing the omitted on the included . The bias term is : it is zero exactly when has no effect on () or when and are uncorrelated (). Clean, but not directly usable, because and both involve the thing you never observed. That is the whole problem, and every method below is a different way around it.

Five ways to put a range on the bias

1. Sensitivity analysis. Instead of assuming no confounding, ask how strong a confounder would need to be to overturn your conclusion. The robustness value captures this: the minimum association (measured as partial ) an unobserved confounder must have with both treatment and outcome to drive the effect to zero. If a weak, implausible confounder is enough to flip the result, the finding is fragile; if only an implausibly strong one would do it, the finding is robust. Cinelli and Hazlett’s contour plots are the standard way to read this off visually, showing where the estimate stays significant across combinations of confounder strength.

2. Bounding approaches. Rather than scan scenarios, fix an assumption about the most explanatory power any omitted variable could plausibly have, and derive hard upper and lower limits on the effect. The bounds are only as credible as that ceiling assumption. Set it too low and the true effect can fall outside your interval; set it too high and the bounds are so wide they say nothing. Covariate benchmarking anchors the assumption in data: argue that no unobserved confounder is stronger than, say, your strongest observed covariate, and use that as the empirical ceiling.

3. Simulation. Build synthetic datasets where you control the omitted variable’s effect, then estimate the treatment effect while deliberately leaving it out, repeatedly. The spread of estimates is the empirical distribution of the bias. Useful for two things: seeing which conditions make OVB worst (stronger confounder-regressor correlation, larger effect on the outcome), and validating that a sensitivity method’s claimed bounds actually cover the truth.

4. Bayesian methods. Put a prior on the unobserved confounder’s parameters (or directly on the bias), and the posterior on the causal effect carries that uncertainty through. You get a full distribution over plausible effects instead of a point estimate. The catch is the obvious one: if the posterior moves a lot when you change the prior, the data isn’t doing the work, your assumptions are.

5. Machine learning. When relationships are non-linear or high-dimensional, flexible models estimate the nuisance functions better than a hand-specified linear model. DoubleML now ships OVB sensitivity analysis inside the double-machine-learning framework, and methods like BART give uncertainty estimates directly. The open problem is the usual one with flexible models: quantifying causal uncertainty rigorously, and explaining what the model is actually conditioning on.

How they compare

What I took away

The honest core of all of it: you cannot measure what you did not observe, so every method here trades the impossible question (“what is the bias?”) for a tractable one (“how strong would a confounder have to be, and is that plausible here?”). Sensitivity analysis and covariate benchmarking are the two I find most useful in practice, because they hand the judgment back to domain knowledge instead of hiding it inside a prior or a simulated data-generating process.

References

The starting points worth reading if you want the real treatment, not my notes:

• Cinelli & Hazlett (2020), Making Sense of Sensitivity: Extending Omitted Variable Bias. The robustness value and contour plots. PDF
• Chernozhukov et al. (2021), Long Story Short: Omitted Variable Bias in Causal Machine Learning. Extends OVB to ML. arXiv
• DoubleML, sensitivity analysis documentation.
• Omitted-variable bias on Wikipedia for the algebra.
• Eggers, Intermediate Causal Inference lecture slides.

More causal inference notes:

• Causal inference cheatsheet
• Assessing overlap in covariate distributions
• Recommendations as treatments

What a mass point is

Say a large number of your observations share the same value, for example a crowd of customers who all bought a $1 product. You log-transform sales to make the distribution more normal, and every one of those $1 values collapses to zero, since . Now the transformed data is dominated by a spike of zeros.

Visually: the original data has a tall spike at , and after the log transform that spike moves to and gets taller relative to everything else.

Why linear models struggle with it

Linear regression leans on a few assumptions:

1. Linearity: the relationship between predictors and outcome is linear.
2. Independence of errors: residuals are independent of each other.
3. Homoscedasticity: residuals have constant variance across all levels of the predictors.
4. Normality: residuals are roughly normally distributed.

A mass point mainly attacks the last two. With a big concentration of identical values (here, the spike at ), two things go wrong:

• Non-normal residuals. The mass point pulls residuals toward itself, introducing asymmetry and distorting their spread, so they no longer sit balanced and normal around zero.
• Heteroscedasticity. Residual variance stops being constant, because the model can’t fit the dense region and the sparse region equally well. That matters because non-constant variance gives you biased standard errors (so t-tests and confidence intervals become unreliable) and inefficient estimates (predictions are less precise than they could be).

The net effect: the model tries to fit a straight line through data that violates its own assumptions, and you get biased or inefficient estimates. On a scatter plot you can see the fitted line bending awkwardly around the cluster at zero.

Models that handle it better

Instead of forcing a single linear fit, use a model that explicitly accounts for the mass point. Three options, with tradeoffs:

Model	Key idea	Pros	Cons	When to use
Hurdle	A classifier first decides whether an observation is in the mass-point group; a separate regression models the continuous values for the rest.	Cleanly separates the spike from the continuous part.	Two models to fit and coordinate.	A mix of a mass point (e.g. lots of or zeros) and continuous data.
Mixture	Model the data as a blend of two distributions: one for the spike, one for the smooth continuous part.	Flexible, captures distinct subpopulations in one model.	Harder to fit; estimating the mixture components can be unstable.	Data that naturally splits into a sharp spike plus a smooth distribution.
Zero-inflated	A logistic component models excess zeros; a continuous model handles the rest.	Purpose-built for far more zeros than a standard model expects.	Best only when the mass point really is at zero; more setup.	An unusually high count of zeros (or one specific mass point).

Takeaway

A giant spike at zero in log-transformed data is a signal, not noise to ignore. Linear regression will fit it, but the standard errors and efficiency you rely on quietly stop being trustworthy. A hurdle, mixture, or zero-inflated model fits the structure of the data instead of fighting it.

Appendix: why heteroscedasticity matters

Effect on the estimates

1. Biased standard errors. Under heteroscedasticity the standard errors of the coefficients are unreliable, which throws off confidence intervals and hypothesis tests and raises the chance of Type I or Type II errors.
2. Inefficient estimates. OLS stays unbiased, but it is no longer the best linear unbiased estimator (BLUE). The estimates have higher variance than necessary, so they are less precise.

Why efficiency is worth caring about

Efficiency means getting estimators with the smallest possible variance. It buys you two things:

1. Precision. Tighter confidence intervals, so the estimates are more reliable.
2. Statistical power. More precision means tests are likelier to detect real effects, lowering the risk of Type II errors.

1. Why do we need to adjust for differences in the covariate distributions?

If there is a region in the covariate space where T has relatively few units/samples or C has relatively few units/samples, our inferences in that region will largely depend on extrapolation, thus will be less credible compared to inferences of regions where both T and C have substantial overlap in the covariates distribution.

Example:
Covariate space #1:

• T has 5 males with age<=18
• C has 55 males with age<=18.

Covariate space #2:

• T has 55 males age>18 and age<=30
• C has 60 males age>18 and age<=30

In the above example, our inferences for the covariate space #2 will be more credible than our inferences for the covariate space #1.

To note, even in cases when there is there is no confounding (unconfoundedness), this is still a fundamental issue. However, if we have a completely randomized experiment, we can expect the distribution of covariates to be similar per definition, and thus, less risks of stumbling upon this issue.

2. Assessing overlap of univariate distributions

1. Comparing differences in location of two distributions

Given two univariate probability distributions and with means and , and variances and , we can estimate the differences in locations of the two distributions with:

, which is a normalized measure of difference of two distributions. To note, this is different from the t-statistic that tests whether the data contain sufficient info to support the hypothesis that two covariate means in T and C are different:

To our purposes, we don’t care about testing if we have enough data to check if means are different between T&C. Instead, we care about understanding if the differences between the distributions are so large that we need to fix the issue, and at the same time, check what kind of adjustment methods are required for the problem at hand.

2. Comparing measures of dispersion of two distributions

The log difference in the standard deviations helps to understand the difference in dispersion:

, we take the log because its more normally distributed compared to simple difference or ratio of standard deviations.

3. Comparing the tail overlaps of two distributions

To understand the overlap of covariate distributions in the tail regions, we can calculate the fraction of treatment units that exist in the tails of the distribution of covariate values in the control.

One method is to choose an arbitrary tail boundary, e.g.  and calculate the probability mass of T’s covariate distribution outside the and quantiles of the C’s covariate distribution:

, where and represent the CDFs of the T and C covariates’ distributions (so is the quantile function of the control distribution). We can do the same for C, where we calculate the probability mass of C’s covariate distribution outside the and quantiles of the T’s covariate distribution:

The intuition here is that it is relatively easy to impute outcomes values of either T or C units in the dense regions of the distribution (e.g. near the mean of a normal distribution where majority of samples are concentrated), while it is harder at the tail regions where only few units exist. One can refer to the following two distributions as an example of two differently dispersed covariates’ distributions:

Last but not least, here’s how to interpret the values:

• happens for a completely randomized experiment, where only of units have covariate values that make us cry (i.e. difficult to impute the missing potential outcomes of a group).
• if , it will be relatively difficult to impute/predict the missing potential outcomes for the control units (more treatment units exist in the tail region of the control group’s covariate distribution).
• if , it will be relatively difficult to impute/predict the missing potential outcomes for the treatment units (more control units exist in the tail region of the treatment group’s covariate distribution).

3. Assessing overlap of multivariate distributions

Given K covariates, we could compare the distributions of T & C iteratively one-by-one using the above approach. A good way is to start with the covariates for which you have a priori belief that it is highly associated with the outcome (i.e. ensuring the importatnt covariates are okay).

4. Estimating if T/C units have comparable twins in C/T

Consider a unit with treatment . For this unit we can determine if there is a comparable twin with treatment such that the difference in propensity scores is less than or equal to , where is an upper threshold implying the difference in propensity scores is less than 10%.

Intuitively, if there is a similar unit with opposite treatment and a similar propensity score, we may be able to obtain trustworthy estimates of causal effects without any extrapolation or additional work. However, if there are many such units without twins, it will be difficult to obtain credible estimates of causal effects, no matter what methods we use. A common method to solve this, given we have enough samples, is to trim the units without twins.

We can estimate the degree of units with close comparison units by defining an indicator variable that flags whether unit has a comparable twin:

, then for each T & C we can define two overlap measures implying the proportion of units with close comparisons:

More causal inference notes:

• Causal inference cheatsheet
• Estimating the distribution of omitted variable bias
• Recommendations as treatments

Core concepts

The assumptions that actually bite

Most causal estimates fall apart not in the math but in one of these three. They are usually untestable, which is exactly why they are worth writing down.

Picking a method: the one thing that matters for each

When I reach for one of these, the question is never “what does it do” but “what does it need to be true.” Here is the load-bearing assumption behind each, the thing that, if violated, quietly makes the estimate wrong.

Things I’ve learned to check before trusting a result

• Plot the overlap. If treated and control don’t share covariate space, the model is extrapolating and nobody told you.
• Stress the key assumption, not the data. More features won’t save a design whose identification is broken.
• Run a sensitivity analysis. “How big would an unobserved confounder have to be to flip this?” is often more honest than the point estimate.
• Be suspicious of a clean answer from messy observational data. It usually means an assumption is doing more work than you realize.

More causal inference notes:

• Assessing overlap in covariate distributions
• Estimating the distribution of omitted variable bias
• Recommendations as treatments

Dot Product

The dot product (also known as the scalar product) of two vectors and in -dimensional space is given by:

This operation results in a single scalar value and provides a measure of the similarity between the two vectors. If the vectors point in the same direction, the dot product is positive and large; if they point in opposite directions, the dot product is negative; if they are orthogonal, the dot product is zero.

Covariance

Covariance is a measure of how much two random variables and change together. It is given by:

Where denotes the expected value. If the covariance is positive, the variables tend to increase together; if it is negative, one tends to increase when the other decreases.

Relationship

The relationship between covariance and the dot product can be seen in the context of centered random variables. If you consider the centered random variables:

the covariance between and can be interpreted as the dot product of these centered variables in the space of random variables, normalized by the number of observations :

Here, and are the deviations from the mean of the random variables and respectively. This sum is analogous to the dot product, where the sum of the products of corresponding elements gives a measure of the overall relationship between the two sets of values.

Summary

In summary, covariance can be viewed as a normalized dot product of centered random variables. Both operations measure how two sets of values relate to each other, with the dot product being a geometric measure in vector space and covariance being a statistical measure in the space of random variables.

Key points:

1. Developing and deploying ML systems might seem fast and cheap, but maintaining it for the long term is difficult and expensive.
2. Goal of dealing with technical debt is not to add new functionality, but to enable future improvements, reduce errors, and improve maintainability.
3. Data influences ML system behavior. Technical debt in ML systems may be difficult to detect because it exists at the system level rather than the code level.

Complex Models Erode Boundaries

Data Dependencies Cost More than Code Dependencies

Feedback Loops

When it comes to live ML systems that update their behavior over time , it is difficult to predict how they will behave when they are released into production. This is called analysis debt - the difficulty of measuring the behavior of a model before deployment.

ML-System Anti-Patterns

In the real world, ML-related code takes a very small fraction of the entire system. Therefore, it is important to consider other parts of the system and make sure that there are as few system-design anti-patterns as possible.

Configuration Debt

ML systems come with various configuration options:
1.  Choice of input features
2.  Algorithm-specific learning configurations (hyperparameters, number of nodes, layers, etc.)
3.  Pre- & post-processing methods
4.  Verification methods

In full-scale systems, configuration code far exceeds the number of lines of traditional code. Mistakes in configuration can lead to loss of time, waste of computing resources, and other production issues. Thus, verifying and testing configurations is crucial.

Example of difficult-to-handle configurations (Choice of input features):
1.  Feature A was incorrectly logged from 9/13-9/17.
2.  Feature B isn’t available before 10/7.
3.  Feature C has to have 2 different acquisition methods due to changes in logging.
4.  Feature D isn’t available in production. Substitute feature D’ must be used.
5.  Feature Z requires extra training memory for the model to train efficiently.
6.  Feature Q has high latency, so it makes Feature R also unusable.

Configurations must be:
1.  Easy to change. It should seem as making a small change to the previous configuration.
2.  Difficult to make manual errors, omissions, or oversights.
3.  Easy to analyze and compare with the previous version.
4.  Easy to check (basic facts and details).
5.  Easy to detect unused or redundant settings.
6.  Code reviewed and maintained in a repository.

Dealing with the Changes in the External World

ML systems often directly interact with the ever-changing real world. Due to volatility in the real world, ML systems require continuous maintenance.

Other Areas of ML-related Debt

Conclusion

1. Speed is not evidence of low technical debt or good practices because the real cost of debt becomes apparent over time.
2. Useful questions for detecting ML technical debt:
   
   • How easily can an entirely new algorithmic approach be tested at full scale?
     
   • What is the transitive closure of all data dependencies? Transitive closure gives you the set of all places you can get to, from any starting place.
     
   • How precisely can the impact of a new change to the system be measured?
     
   • Does improving one model or signal degrade others?
     
   • How quickly can new members of the team be brought up to speed?
3. Key areas that will help reduce technical debt:
   
   • Maintainable ML
     
   • Better abstractions
     
   • Testing methodologies
     
   • Design patterns
4. Engineers and researchers both need to be aware of technical debt. “Research solutions that provide a tiny accuracy benefit at the cost of massive increases in system complexity are rarely wise practice.”
5. Dealing with technical debt can only be achieved by a change in team culture. “Recognizing, prioritizing, and rewarding this effort is important for the long term health of successful ML teams.”

And this is it!

In this post, I have tried to summarize the key points of the prominent paper “Hidden Technical Debt in Machine Learning Systems” by Google.

Contents

1. Anonymized data set know-how
2. Cross-validation know-how
3. Neural network optimization with Keras-tuner
4. Keras fast inference know-how

Anonymized data set know-how

Since we have been provided with an anonymized data set, it was difficult to perform EDA.

The specific data engineering methodology of Jane Street is one of their primary assets and that is why they have anonymized their data set. Hence, I didn’t spend much time on de-anonymization, considering it was unethical.

However, for those who are interested in how to start de-anonymizing a data set, here are example de-anonymization notebooks shared by Gregory Calvez during the competition:

1. https://www.kaggle.com/gregorycalvez/de-anonymization-buy-sell-net-gross
2. https://www.kaggle.com/gregorycalvez/de-anonymization-price-quantity-stocks
3. https://www.kaggle.com/gregorycalvez/de-anonymization-time-aggregation-tags
4. https://www.kaggle.com/gregorycalvez/de-anonymization-min-max-and-time

The author of the notebook explains his mindset throughout the notebook in clear and concise comments.

Cross-validation know-how

We have been given 2 years of time-series data.

To avoid information leakage, we couldn’t randomly split the data into cross-validation splits.

The main cross-validation methodology used in the competition was referred to as “Purged Group Time Series Split”. Here is how we split the data:

The intuition is that we iteratively expand the training data size over the entire history of a time series and repeatedly test against a forecasting window, without dropping older data points (Learn more about the method here).

Here is my implementation of the “Purged Group Time Series Split”:

import pandas as pd
import numpy as np

def custom_cv_split(df, n_splits=4, date_column_name):
    date_splits = np.array_split(df[date_column_name].unique(), n_splits)
    train_ixs = []
    test_ixs = []
    for split in date_splits[:-1]:
        if len(train_ixs) > 0:
            curr_tr_ix = train_ixs[-1].copy()
            curr_tr_ix.extend(df[df[date_column_name].isin(split)].index.values.tolist())
        else:
            curr_tr_ix = df[df[date_column_name].isin(split)].index.values.tolist()
        train_ixs.append(curr_tr_ix)
    for split in date_splits[1:]:
        curr_ts_ix = df[df[date_column_name].isin(split)].index.values.tolist()
        test_ixs.append(curr_ts_ix)
        
    for i in train_ixs:
        print("Training size: ", len(i), i[:10], i[-10:])
    print()    
    for i in test_ixs:
        print("Test size: ", len(i), i[:10], i[-10:])
    return list(zip(train_ixs, test_ixs))

If you face any bugs, have fun debugging it. I am sure it will help you understand the code!

Neural network optimization with Keras-tuner

Keras-tuner package has made it easy and intuitive to tune neural network parameters! Their documentation is also quite simple.

However, when it comes to using custom cross-validation sets, we have to apply a small trick by creating and modifying the keras-tuner “kt.engine.tuner.Tuner” class:

class CVTuner(kt.engine.tuner.Tuner):
    def run_trial(self, trial, X, y, splits, batch_size=32, epochs=1,callbacks=None):
        val_losses = []
        for train_indices, test_indices in splits:
            X_train, X_test = [x[train_indices] for x in X], [x[test_indices] for x in X]
            y_train, y_test = [a[train_indices] for a in y], [a[test_indices] for a in y]
            if len(X_train) < 2:
                X_train = X_train[0]
                X_test = X_test[0]
            if len(y_train) < 2:
                y_train = y_train[0]
                y_test = y_test[0]
            
            model = self.hypermodel.build(trial.hyperparameters)
            hist = model.fit(X_train,y_train,
                      validation_data=(X_test,y_test),
                      epochs=epochs,
                        batch_size=batch_size,
                      callbacks=callbacks)
            
            val_losses.append(np.max([hist.history[k]) for k in hist.history]) #change this based on your metric, we use auc --> so we want to get maximum value
        val_losses = np.asarray(val_losses)
        self.oracle.update_trial(trial.trial_id, {k:np.mean(val_losses[:,i]) for i,k in enumerate(hist.history.keys())})
        self.save_model(trial.trial_id, model)

Make sure you understand each line, so that you can customize it later for your own use case.

As a next step, we can directly use our “CVTuner” class as:

tuner = CVTuner(
    hypermodel=model_fn, #keras model definition function
    oracle=kt.oracles.BayesianOptimization(
        objective= kt.Objective('val_loss', direction='min'),
        num_initial_points=4,
        max_trials=10,
        seed=SEED
        )
)
        
tuner.search(
    (X,), 
    (X,y), 
    splits=splits, #here we pass the CV folds
    batch_size=4096,
    epochs=100,
    callbacks=[EarlyStopping('val_loss',patience=5)]
)

And that is it!

With the help of the keras-tuner package, we can setup the hyperparameter tuning in only a few lines of code.

Keras fast inference know-how

To simulate a real-world High-Frequency-Trading (HFT) prediction scenario, the organizers have set a time limit of ~60 predictions/second.

This made heavy feature engineering almost impossible, leading us to focus on methods that improve the prediction/second rate.

One of the interesting methods was to make our Keras NN model inference time ~3-4 times faster by making our model “LiteModel”:

class LiteModel:
    
    @classmethod
    def from_file(cls, model_path):
        return LiteModel(tf.lite.Interpreter(model_path=model_path))
    
    @classmethod
    def from_keras_model(cls, kmodel):
        converter = tf.lite.TFLiteConverter.from_keras_model(kmodel)
        tflite_model = converter.convert()
        return LiteModel(tf.lite.Interpreter(model_content=tflite_model))
    
    def __init__(self, interpreter):
        self.interpreter = interpreter
        self.interpreter.allocate_tensors()
        input_det = self.interpreter.get_input_details()[0]
        output_det = self.interpreter.get_output_details()[0]
        self.input_index = input_det["index"]
        self.output_index = output_det["index"]
        self.input_shape = input_det["shape"]
        self.output_shape = output_det["shape"]
        self.input_dtype = input_det["dtype"]
        self.output_dtype = output_det["dtype"]
        
    def predict(self, inp):
        inp = inp.astype(self.input_dtype)
        count = inp.shape[0]
        out = np.zeros((count, self.output_shape[1]), dtype=self.output_dtype)
        for i in range(count):
            self.interpreter.set_tensor(self.input_index, inp[i:i+1])
            self.interpreter.invoke()
            out[i] = self.interpreter.get_tensor(self.output_index)[0]
        return out
    
    def predict_single(self, inp):
        """ Like predict(), but only for a single record. The input data can be a Python list. """
        inp = np.array([inp], dtype=self.input_dtype)
        self.interpreter.set_tensor(self.input_index, inp)
        self.interpreter.invoke()
        out = self.interpreter.get_tensor(self.output_index)
        return out[0]

And finally, transform our model using the “TFLite” class:

model = LiteModel.from_keras_model(model)
model.predict(X_test)

By doing this simple step, we have been able to drastically increase our predictions/second. Approximately 3-4 times!!!

However, one thing to note here is that since “TFLite” is an optimization method, there might be slight differences in the performance of the transformed model.

Despite this withdrawal, according to my experiments, I did not see a huge drop in the performance, so I used “TFLite” transformation without worrying too much.

Conclusion

To conclude, I have learned many valuable techniques by participating in the “Jane Street Market Prediction” competition.

The winners of the competition will be announced after 6 months. Until then, throughout these 6 months, our models are tested against real-time market data.

Ultimately, no matter if you win a medal or not, participating in Kaggle competitions is surely the best way to acquire up-to-date State Of The Art (SOTA) ML modeling knowledge!

Therefore, for anyone who is interested in Data Science, Machine Learning or AI in general, I highly encourage you to try to Kaggle.

Thank you for reading and happy Kaggling!

1. How to train a model when only 1% of the samples are positive samples?

Before diving deep, let’s train a classifier and calculate our baseline scores!

First, let’s create a simple data set:

import pandas as pd
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=100000, 
                           n_features=30, 
                           n_redundant=0,
                           n_clusters_per_class=2, 
                           weights=[0.98], 
                           flip_y=0, 
                           random_state=12345)

By setting weights to 0.98, we are creating an imbalanced data set

For ease of use, let’s convert the arrays into a DataFrame:

cols = ["feature_"+str(i) for i in range(1, X.shape[1]+1)]
df = pd.DataFrame.from_records(X, columns=cols)
df["Class"] = y

Let’s split the data into train/test sets to evaluate our models on a common test set:

pos_samples = df[df["Class"]==1].sample(frac=1) #extracting positive samples
neg_samples = df[df["Class"]==0].sample(frac=1) #extracting negative samples

# let's set aside 500 positive and 500 negative samples
test = pd.concat((pos_samples[:500], neg_samples[:500]), axis=0).sample(frac=1).reset_index(drop=True) #combine, shuffle, reset indices

# let's use the rest for training
train = pd.concat((pos_samples[500:], neg_samples[500:]), axis=0).sample(frac=1).reset_index(drop=True) #combine, shuffle, reset indices

Now let’s train a model using the training set and evaluate on the test set (for simplicity, I am going to use sklearn’s “RandomForestClassifier”):

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=1000, #the more the better, but slower
                               random_state=12345, #lucky number
                               class_weight="balanced",
                               verbose=2,
                               n_jobs=-1).fit(train.values[:,:-1], train["Class"])

Make sure, the “class_weight” parameter is set to “balanced”, since we are dealing with an imbalanced classification problem!

Since we are working with an imbalanced data, let’s use sklearn’s “balanced_accuracy_score”, “classification_report” and “plot_confusion_matrix” functionalities to evaluate our model:

from sklearn.metrics import classification_report, balanced_accuracy_score, plot_confusion_matrix

preds = model.predict(test.values[:,:-1])
print(classification_report(test["Class"], preds))
print("Accuracy: {}%".format(int(balanced_accuracy_score(test["Class"], preds)*100)))

              precision    recall  f1-score   support

           0       0.74      1.00      0.85       500
           1       1.00      0.64      0.78       500

    accuracy                           0.82      1000
   macro avg       0.87      0.82      0.82      1000
weighted avg       0.87      0.82      0.82      1000

Accuracy: 82%

import matplotlib.pyplot as plt
plt.rcParams.update({'font.size': 15})

fig, ax = plt.subplots(figsize=(5, 5))
_=plot_confusion_matrix(model, test.values[:,:-1], test["Class"], values_format = '.0f', cmap=plt.cm.Blues, ax=ax)

Bagging

What is Bagging?
Bagging is a sampling method, commonly used in ensemble learning. It means splitting the training data into several subsets and training a model on each subset. After training the models, each model generates a prediction for a sample and all predictions are averaged to produce the prediction.

How to choose the number of splits?
There is no set value for the number of splits. However, based on how many positive samples there are, I usually choose 5-10 subsets. (Split as long as there is improvement)

pos_samples = train[train["Class"]==1].sample(frac=1)
neg_samples = train[train["Class"]==0].sample(frac=1)

#lets split into 5 bags
train_1 = pd.concat((pos_samples[:300], neg_samples[:3000]), axis=0)
train_2 = pd.concat((pos_samples[300:600], neg_samples[3000:6000]), axis=0)
train_3 = pd.concat((pos_samples[600:900], neg_samples[6000:9000]), axis=0)
train_4 = pd.concat((pos_samples[900:1200], neg_samples[9000:12000]), axis=0)
train_5 = pd.concat((pos_samples[1200:], neg_samples[12000:15000]), axis=0)

Train 5 models for 5 splits:

bag_1 = RandomForestClassifier(n_estimators=1000, random_state=12345, class_weight="balanced", n_jobs=-1).fit(train_1.values[:,:-1], train_1["Class"])
bag_2 = RandomForestClassifier(n_estimators=1000, random_state=12345, class_weight="balanced", n_jobs=-1).fit(train_2.values[:,:-1], train_2["Class"])
bag_3 = RandomForestClassifier(n_estimators=1000, random_state=12345, class_weight="balanced", n_jobs=-1).fit(train_3.values[:,:-1], train_3["Class"])
bag_4 = RandomForestClassifier(n_estimators=1000, random_state=12345, class_weight="balanced", n_jobs=-1).fit(train_4.values[:,:-1], train_4["Class"])
bag_5 = RandomForestClassifier(n_estimators=1000, random_state=12345, class_weight="balanced", n_jobs=-1).fit(train_5.values[:,:-1], train_5["Class"])

How to combine the predictions of the models?
We can use the “predict_proba” function of our “RandomForestClassifier” model to get the probabilities of each sample belonging to a specific class!

probs_1 = bag_1.predict_proba(test.values[:,:-1])[:,1]
probs_2 = bag_2.predict_proba(test.values[:,:-1])[:,1]
probs_3 = bag_3.predict_proba(test.values[:,:-1])[:,1]
probs_4 = bag_4.predict_proba(test.values[:,:-1])[:,1]
probs_5 = bag_5.predict_proba(test.values[:,:-1])[:,1]

Let’s evaluate our models:

probs = (probs_1+probs_2+probs_3+probs_4+probs_5)/5
preds = [1 if prob >= 0.5 else 0 for prob in probs]
print(classification_report(test["Class"], preds))
print("Accuracy: {}%".format(int(balanced_accuracy_score(test["Class"], preds)*100)))

              precision    recall  f1-score   support

           0       0.76      1.00      0.87       500
           1       1.00      0.69      0.82       500

    accuracy                           0.85      1000
   macro avg       0.88      0.85      0.84      1000
weighted avg       0.88      0.85      0.84      1000

Accuracy: 84%

from sklearn.metrics import confusion_matrix
import seaborn as sns

cm = confusion_matrix(test["Class"], preds)
ax= plt.subplot()
sns.heatmap(cm, annot=True, ax = ax, fmt='g', cmap=plt.cm.Blues)

# labels, title and ticks
ax.set_xlabel('Predicted labels');ax.set_ylabel('True labels')
ax.xaxis.set_ticklabels(['0', '1']); ax.yaxis.set_ticklabels(['1', '0'])
_=plt.tight_layout()

2. How to choose the decision threshold of a model?

The advantage of the Bagging method is observed when we choose optimal decision thresholds for our classifiers.

But what is a decision threshold?
For example, in a binary classification problem with class labels 0 and 1, with predicted probabilities and a decision threshold of 0.5, the predicted probabilities less than the threshold of 0.5 are assigned to class 0 and values greater than or equal to 0.5 are assigned to class 1.

• Prediction < 0.5 = Class 0
• Prediction >= 0.5 = Class 1

Why do we need to optimize the decision thresholds?
https://stats.stackexchange.com/questions/312119/reduce-classification-probability-threshold

Okay let’s start!

dts = [i/100 for i in range(10, 100, 5)]
accs = []
for dt in dts:
    probs = model.predict_proba(test.values[:,:-1])[:,1]
    preds = [1 if prob >= dt else 0 for prob in probs]
    acc = balanced_accuracy_score(test["Class"], preds)
    accs.append(acc)

fig=plt.figure(figsize=(12, 5))
_=plt.plot(accs)
_=plt.xticks([i for i in range(len(dts))], dts)
_=plt.grid()
_=plt.tight_layout()
_=plt.xlabel("Decision thresholds")
_=plt.ylabel("Accuracies")

probs = model.predict_proba(test.values[:,:-1])[:,1]
preds = [1 if prob >= 0.1 else 0 for prob in probs]
print(classification_report(test["Class"], preds))
print("Accuracy: {}%".format(int(balanced_accuracy_score(test["Class"], preds)*100)))

              precision    recall  f1-score   support

           0       0.83      0.99      0.91       500
           1       0.99      0.80      0.89       500

    accuracy                           0.90      1000
   macro avg       0.91      0.90      0.90      1000
weighted avg       0.91      0.90      0.90      1000

Accuracy: 89%

cm = confusion_matrix(test["Class"], preds)
ax= plt.subplot()
sns.heatmap(cm, annot=True, ax = ax, fmt='g', cmap=plt.cm.Blues)
ax.set_xlabel('Predicted labels')
ax.set_ylabel('True labels')
ax.xaxis.set_ticklabels(['0', '1']); ax.yaxis.set_ticklabels(['1', '0'])
_=plt.tight_layout()

dts = [i/100 for i in range(10, 100, 5)]
accs = []
for dt in dts:
    probs_1 = bag_1.predict_proba(test.values[:,:-1])[:,1]
    probs_2 = bag_2.predict_proba(test.values[:,:-1])[:,1]
    probs_3 = bag_3.predict_proba(test.values[:,:-1])[:,1]
    probs_4 = bag_4.predict_proba(test.values[:,:-1])[:,1]
    probs_5 = bag_5.predict_proba(test.values[:,:-1])[:,1]
    probs = (probs_1+probs_2+probs_3+probs_4+probs_5)/5
    preds = [1 if prob >= dt else 0 for prob in probs]
    acc = balanced_accuracy_score(test["Class"], preds)
    accs.append(acc)

fig=plt.figure(figsize=(12, 5))
_=plt.plot(accs)
_=plt.xticks([i for i in range(len(dts))], dts)
_=plt.grid()
_=plt.tight_layout()
_=plt.xlabel("Decision thresholds")
_=plt.ylabel("Accuracies")

probs_1 = bag_1.predict_proba(test.values[:,:-1])[:,1]
probs_2 = bag_2.predict_proba(test.values[:,:-1])[:,1]
probs_3 = bag_3.predict_proba(test.values[:,:-1])[:,1]
probs_4 = bag_4.predict_proba(test.values[:,:-1])[:,1]
probs_5 = bag_5.predict_proba(test.values[:,:-1])[:,1]
probs = (probs_1+probs_2+probs_3+probs_4+probs_5)/5
preds = [1 if prob >= 0.1 else 0 for prob in probs]
print(classification_report(test["Class"], preds))
print("Accuracy: {}%".format(int(balanced_accuracy_score(test["Class"], preds)*100)))

              precision    recall  f1-score   support

           0       0.88      0.96      0.92       500
           1       0.96      0.86      0.91       500

    accuracy                           0.91      1000
   macro avg       0.92      0.91      0.91      1000
weighted avg       0.92      0.91      0.91      1000

Accuracy: 91%

cm = confusion_matrix(test["Class"], preds)
ax= plt.subplot()
sns.heatmap(cm, annot=True, ax = ax, fmt='g', cmap=plt.cm.Blues)
ax.set_xlabel('Predicted labels')
ax.set_ylabel('True labels')
ax.xaxis.set_ticklabels(['0', '1']); ax.yaxis.set_ticklabels(['1', '0'])
_=plt.tight_layout()

3. How to evaluate a model, so that the client can easily understand the capabilities of the model?

Last, but not least, let’s talk about the interpretability of our results.

It is common for our clients to not have prior knowledge of ML evaluation methods. However, they would like to know if our models are robust or not.
How do we solve this problem?

For binary classification problems, here is a solution:
In the production environment, our classifier is most likely to see different types of data with different distributions per class. Therefore, we can evaluate the capability of our models by simulating different input distributions!

Example:

from sklearn.metrics import accuracy_score
from collections import Counter
import numpy as np

init=1
rows = []
for i in range(11):
    neg = test[test["Class"]==0].reset_index(drop=True)
    pos = test[test["Class"]==1].reset_index(drop=True)
    neg_sample = neg[:int(neg.shape[0]*init)]
    pos_sample = pos[:int((((neg.shape[0]*init)*1)/init)*(1-init))]
    combined = pd.concat((neg_sample, pos_sample), axis=0)
    probs_1 = bag_1.predict_proba(combined.values[:,:-1])[:,1]
    probs_2 = bag_2.predict_proba(combined.values[:,:-1])[:,1]
    probs_3 = bag_3.predict_proba(combined.values[:,:-1])[:,1]
    probs_4 = bag_4.predict_proba(combined.values[:,:-1])[:,1]
    probs_5 = bag_5.predict_proba(combined.values[:,:-1])[:,1]
    probs = (probs_1+probs_2+probs_3+probs_4+probs_5)/5
    preds = [1 if prob >= 0.1 else 0 for prob in probs]
    acc = accuracy_score(preds, combined["Class"])
    neg_preds = (Counter(preds)[0]/len(preds))*100
    pos_preds = (Counter(preds)[1]/len(preds))*100
    rows.append([np.ceil(init*100), np.ceil((1-init)*100), neg_preds, pos_preds, acc])
    init = init - 0.1
    
df = pd.DataFrame.from_records(rows, columns=["Actual - distribution", "Actual + distribution", "Predicted - distribution", "Actual + distribution", "Accuracy"])

Let’s take a look at the results:

X_test = test[cols]
y_test = test["Class"]

from sklearn.model_selection import KFold

cv = KFold(n_splits=10, random_state=12345, shuffle=True)
accs = []
for train_index, test_index in cv.split(X_test, y_test):
    xtest, ytest = X_test.iloc[test_index], y_test.iloc[test_index]
    probs_1 = bag_1.predict_proba(xtest)[:,1]
    probs_2 = bag_2.predict_proba(xtest)[:,1]
    probs_3 = bag_3.predict_proba(xtest)[:,1]
    probs_4 = bag_4.predict_proba(xtest)[:,1]
    probs_5 = bag_5.predict_proba(xtest)[:,1]
    probs = (probs_1+probs_2+probs_3+probs_4+probs_5)/5
    preds = [1 if prob >= 0.15 else 0 for prob in probs]
    acc = balanced_accuracy_score(ytest, preds)
    accs.append(acc)
    
accs_ = np.array(accs) 
print("Accuracy: %0.2f (+/- %0.2f)" % (accs_.mean(), accs_.std() * 2))   

Accuracy: 0.91 (+/- 0.05)

Conclusion:

1. In this post, I have tried to introduce an ensemble sampling method called “Bagging”
2. In the beginning, it was difficult to assess the advantage of using the Bagging method. However, once we have optimized our decision thresholds, we were able to see the benefits
3. It is common for our clients to require interpretable evaluation metrics. Therefore, I have shared a simple evaluation method for explaining classification models