A Problem With Segmentation (With Simulation)

Posted onOctober 1, 2018 Edit onAugust 7, 2021 by Christopher Berry

My contemporaneous notes from a particular INFORMS Marketing Science Conference six years ago feature the letters W, T, and F scrawled in the margins a few times. I learned of a deeper problem lurking in the way we were using the crosstab to identify segmentation. In this post, I’ll unpack a heap of jargon and lay the concern bare.

To the twenty or so marketing scientists in the room at the time, I read concern on the faces of about a dozen. It was a atypical because typically that community doesn’t get concerned about too much. One leader remarked that most in industry were not even executing basic segmentation on their users, so it wasn’t a huge industrial concern, but for marketing scientists in academia, this could be a very bad problem. For industry, for market researchers, it was about reporting basic univariate means – that 22% of consumers would buy purple paper towel, or that Obama would win 51% of the popular vote – were no problem and where the action was. Segmenting consumers, that 55% of women with dogs bought paper towel, was a different matter. Another gentleman, one of the four grandees I knew from the era, repeated, as though to wave everybody else off from the problem, that somebody he had supervised in the past was working the underling problem.

If the presentation ever made it to publication, I can’t find it. And I’m having a very tough time locating the specific author, or if there was ever just a simple resolution.

The problem originates with the way we understand ground truth. So let’s start there an unpack it all.

Assume a Universe. This Universe exists entirely within the RAM of my machine. Or yours. And in this Universe, I create a hundred thousand humans. And I assert two things about each of them. Attribute A and Action B. Either somebody has Attribute A, or they don’t. They either have a 1, or a 0. And they will imminently do Action B. They will either do B, a 1, or they won’t, a 0.

If it helps you to think about Attribute A in a concrete term, you can associate it age. Either somebody has accrued 35 years on Earth-RAM, or they have not. They are either 35 years old, a 1, or they are not. And if helps you to think about Action B, you can think of it as they will either Buy brand B, a 1, or they will not, a 0. It doesn’t really matter. A is a 1 or a 0. And B is a 1 or a 0.

Because I have created this Universe, there are no missing values. There are no import errors. There is no number 2. Or -1. There are certainly no special characters.

There is only cold, sterile, dichotomous, data.

This data, of 100,000 perfect people in a perfect bank of RAM in a machine is the absolute ground truth of the Universe. Many of you may feel bad for those people, but that’s because you’re crazy, they have no feelings, and their existence in this Universe is much better. This is what I tell the computer to create this Universe.

n = 100000
dict_a = {a: np.random.randint(0,2) for a in range(n)}
dict_b = {b: np.random.randint(0,2) for b in range(n)}
df = pd.DataFrame({'A':dict_a, 'B':dict_b}, columns=list('AB'))

As a God, I can run a query and discover the true, actual, ground truth of the Universe at any time. This forms the basis of all Judgement to come. And while I’m playing God, I’m going to instantiate twenty special agents – and task them with a few challenges.

First, I’m going to ask these agents to estimate, out of the 100,000 humans, do a majority of humans have Attribute A? How many have Attribute A?

Since I’m also a frugal God, and not much into apotheosis, I’m not just going to give these twenty agents read access to the entire human database. I’ll give them the ability to sample it. They may have a slice to check for the attribute.

sample_size = 100
agents = 20
i = 0
sample_predicted_values = {}
    while i < agents:
        df_sample = df.iloc[random.sample(range(1, n), sample_size)]
        sample_predicted_values[i] = df_sample['A'].mean()
        i += 1
df_samples = pd.DataFrame.from_dict(sample_predicted_values, orient='index')
print df_samples

These agents are able to sample at random – one may be able to interview human 99, 107, 998…and so on. One may coincidentally interview humans 998,000 to 998,999. Their interview list is generated randomly and independently from each other. The humans in RAM can’t refuse to respond. They have to respond. And they have to respond honestly. And they do respond. They get to ask 100 humans what they are. The agents do not make a mistake in recording the responses or tabulating means. They have no feelings about what the answer should be.

What I’ve set up here is a simulation. It’s a controlled environment mostly free from most of the problems caused by the real Universe out there. How would twenty agents answer the question, how many humans have Attribute A. Put a different way, how many humans in the Universe have a 1 under column A?

They’d probably add up all the ‘1” they got from those 100 humans and apply that average to the 100,000. It would stand to reason that if they asked 100 humans, and 50 of them had attribute A, that 50% of all humans would have attribute A.

What do you think these twenty agents would come back with? Would they all come back with the same answer? Would they all literally say 50%. How probable do you that is?

That probably depends on the underlining distribution. What if the randomness of the birth of this Universe produced more than 50% with Attribute A? Or less than 50%? Wouldn’t one expect np.random, or the random function contained in the numpy library, a random function, to produce something that isn’t perfectly 50%?

It would also depend on the random sample that the agent is pulling too. They’re polling 100 humans from a random list they, themselves, produce at random. And what are the odds that each agent, operating independently of each other, is going to draw the exact same 100 humans to poll? It really isn’t likely.

These twenty agents are going to return different lists of 1’s and 0’s, 100 items long, and arrive at a different percentage estimate of the ground truth. And remember, nobody, other than the God, which is me, knows the real ground truth.

Here’s how my agents did on the first run of the Universe. It’s a sorted list of what each Agent reported back, as a decimal. If it says 0.42 in the table, it can be interpreted as 42%. I sorted the list to make it easier to read. Below, you can see that Agent 3 predicted 42%. Agent 7, and Agent 9 both predicted 50%. Agent 4 predicted 60%.

That’s a total range of 18 percentage points!

Isn’t that interesting? It isn’t that Agent 4 is a bad person, any more than Agent 3 is a bad person. They just happened to generate very different lists, from the same Universe, and arrived at very different answers. And, you don’t know for sure if Agent 4 is right. Or if Agent 3 is right.

You might be eyeballing that list, and your gut might be telling you something. The agents, in aggregate, are returning a distribution of answers that are clustering around some figure in the middle, aren’t they? And maybe you reach for those classic summary statistics, the mean and the median, to describe the list. Maybe you take the average of the averages and end up at 49.75%. Maybe you take the median at 50%? All of these estimates, in total, are clumping near the middle.

It turns out, in this instance, the ground truth in this Universe was that 50,058 humans had Attribute A, so the Ground Truth was 50.058%. Why would God allow such a Universe to exist?!? God plays dice.

Let’s spool up another Universe, and this time, let’s loosen the strings up a bit on our Agents. Let’s gift them ten times the information. Let’s grant them the RAM to ask 1,000 random humans. What then?

Two very interesting things happen. For one, a set of 1000 allows for more precision in their estimates. And, second, the range is a lot tighter, just 4.5% points separate Agent 2 from Agent 13. So based off these twenty estimates, what would you say the ground truth is? Is Agent 0 right at their 50.0% estimate? Is Agent 7 at 50.4%? The mean and the median are very close to each other, around 0.504, or 50.4%. Do you feel more confident that more than half of the humans in this Universe have Attribute A?

The actual ground truth, in this example, was that 49,677 out of 100,000 humans had Attribute A. Agent 11 was more right than others. And again, just out of pure chance.

Did you feel the precision in the figures give you a false sense of confidence? Did you feel that? Insidious, isn’t?

Let’s do one more.

I’m going to give these agents a sample size of 10,000! Such generosity!

#set size of universe
n = 100000
#set size of the sample to be pulled
sample_size = 10000

Each agent is more precise. And the range went from 0.045 to 0.0136, (4.5% down to 1.36%). Which is a bit disappointing isn’t, since I increased the the amount of sample given to each agent by a factor of 10 and only got a four factor increase in accuracy.

And all twenty agents are estimating figures far away from 50%, they’re all down at around 20%. Is Agent 3 right, at 19.93%? Is it weird that Agent 4 and Agent 8 both agree so much at 20.00%? The average of the average is right around Agent 3, 4, and 8 at 0.199, or 19.9%.

Could they all be so wrong?

In this Universe, 19956 humans had Attribute A. The right answer, the actual ground truth of the Universe, was 19.956%

Why?

dict_a = {a: np.random.choice(np.arange(0, 2), p=[0.80, 0.20]) for a in range(n)}

Because God microwaved the dice. I set p=[0.80, 0.20], manipulated the plastic in those bad boys, causing the dice to bounce a certain way – giving the odds a 20% chance of Attribute A for each human. But there was still chance at work.

You may have anchored on 0.50, so when you saw so many outliers, you may have thought there was something wrong with the agents.

Alright, so now you have some intuition for how simulation can help you understand more about the wonderful world of univariate statistics and making predictions about the ground truth of the Universe.

Controlled Universe. Controlled Agents. One variable. One prediction. Why we call it univariate (uni = one) analysis. And there’s still quite a bit of randomness.

As the amount of information made available to each agent increased, the accuracy of their predictions became better. They got better, even if it didn’t get as better as the size of data they got. I increased the amount of data they had access to by two orders of magnitude, from 100 to 10,000, and they only got better from being 9 percentage points off (18 was the total range, the agents were within 9 points of the mean) to almost a one percentage point off. And the rate that predictions get better, themselves, generalize into a predictable formula. Mathematical scientists have in turn discovered a whole bunch of observations about the way this relationship works, assigned Greek letters to some of the concepts, and have been working ever since.

This is the intuition at the root of how nature, of how randomness and chance, even against a ground truth, generalize out in the world of univariate analysis. But because we come to know nature, we can manage that, and ourselves.

Bivariate

Now we’re going to compare two things. I’m spooling up a Universe and setting A and B.

n = 100000
sample_size = 100
dict_a = {a: np.random.randint(0,2) for a in range(n)}
dict_b = {a: np.random.randint(0,2) for a in range(n)}

And now I’m going to ask my twenty agents – how many of those with Attribute A are going to do Action B?

Stated a slightly different way, how many humans have a 1 in both columns, A and B?

I’m going to start off by giving each agent 100 humans to interview. Same rules apply as above.

Here’s what Agent 6 put together from their sample.

Agent
6
Sample Crosstab
A     0   1  All
B               
0    27  37   64
1    19  17   36
All  46  54  100

Here’s how to read that crosstab (sometimes it’s called a contingency table). Agent 6 interviewed 100 people. Of them, 54 had attribute A. And 36 did Action B. Agent 6 found that 17 people that had attribute A, also did Action B.

I’m reporting each agent’s count of those that had 1’s in each column, 1 for Attribute A, and a 1 for Action B.

Sorted Values 1_1
     0
6   17
1   18
11  18
16  19
17  20
15  20
0   21
18  22
2   23
4   23
12  23
7   24
8   24
9   25
3   26
13  26
19  26
14  30
5   33
10  38
Range of Values 1_1
0    21

And as you can see, agents would estimate a value as low as 17 (Agent 6) to as high as twice that, 38 (Agent 1)! It’s trivial here to put those into percentages – a range that varies from 17% to 38%.

The ground truth figure was 25,180, (or 25.180% had both Attribute A and did Action B).

Universal CrossTab
A        0      1     All
B                        
0    24863  24906   49769
1    25051  25180   50231
All  49914  50086  100000

So what happens when I give the agents a sample of 1000?

#set size of the sample to be pulled
sample_size = 1000

And we find that the agents are returning a tighter range, which is what we expected.

Sorted Values 1_1
      0
0   219
16  224
11  236
17  241
14  241
13  241
9   241
10  242
12  248
18  249
8   250
1   250
15  251
7   254
6   255
5   258
3   261
19  261
2   267
4   270
Range of Values 1_1
0    51

And it tightens. Agent Zero estimates 21.9%, Agent Four estimates 27.0%. Which are much tighter in against the ground truth, which was 25.202%.

Universal CrossTab
A        0      1     All
B                        
0    24996  24858   49854
1    24944  25202   50146
All  49940  50060  100000

So what happens when I give the agents a sample of 10000?

#set size of the sample to be pulled
sample_size = 10000

And we find that the estimates are narrowed again.

Sorted Values 1_1
       0
8   2417
12  2426
2   2427
10  2436
15  2454
5   2464
18  2476
19  2486
9   2489
7   2489
0   2525
14  2530
1   2530
3   2538
13  2538
4   2543
6   2546
11  2546
17  2560
16  2570
Range of Values 1_1
0    153

Which is good, against the ground truth of the situation.

Universal CrossTab
A        0      1     All
B                        
0    25197  24885   50082
1    24772  25146   49918
All  49969  50031  100000

So as you can see, more information helps the agents greatly in making a more accurate estimate of the ground truth.

So far, so good.

Next, I’m going to ask my agents a different question – what is the relationship between Attribute A, and Action B?

So I recreate the Universe, and give every agent a sample of 100.

#set size of the sample to be pulled
sample_size = 100

Here’s what Agent 17 discovered when they went out into the Universe.

Agent
17
Sample Crosstab
A     0   1  All
B               
0    26  25   51
1    26  23   49
All  52  48  100

So what can deduce about the Ground Truth of the Universe from this distribution? Well perhaps not much. If Attribute A had nothing to do with Action B, one might expect there to be a number around 25 in each cell. The number of people with Attribute A and doing Action B would be around 25 (And down below, I’ll report out the expected values). And we look above, and we can see that the Agent can’t really make a prediction using information about A.

If Agent 17 was sitting in a room, and I sent a human at random in there, and the person said if they had Attribute A, the Agent might guess that they were going to do Action B in the future (23/48 = ~48%). But given how small the numbers are, Agent 17 couldn’t really say for sure. They wouldn’t be very confident in that prediction.

Agent 5 would have a different relationship with reality based on their experience.

Take a look at what Agent 5 discovered.

Agent
5
Sample Crosstab
A     0   1  All
B               
0    30  18   48
1    21  31   52
All  51  49  100

Agent 5 found that if somebody had Attribute A, 31 times out of 49, that human also did Action B (63% of the time). That’s pretty lopsided. So if you sent a random human into a room with Agent 5, and that human said they had Attribute A, Agent 5 would estimate they’d have a 63% chance that the human would do Action B. They might be pretty confident about that prediction.

That relationship, between a prediction and the confidence in it is quantified in a bunch of equations, rooted in observations, and expressed in Greek letters. I’m going to give them the gift of knowing about a Greek Letter called Chi, and have them report what they calculate.

The way Chi Squared is calculated is by taking the difference between what the agent observes in each cell in the crostab table, and what the agent would expect to see if there was no relationship between A and B, and then square it. Then they divide that number by what was expected. Add all of these together and one gets the Chi Squared value. (The Greek letter Chi looks like a Latin X).

For intuition, the greater the excursion from what’s expected, the greater the Chi Squared value.

The relationship between the Chi Squared value and the probability that it’s independent is moderated by a concept called the degrees of freedom. We have a 2×2 crosstab table here, so the total number of degrees of freedom is just 1. Just one degree of freedom. Check out the black line, the one furthest to the bottom of the chart, below.

As the excursion from what’s expected increases, the Chi Square value returned from the calculation increases. As the Chi Square value increases, the P-value, or the probability that there is no relationship between the two, goes down. High departures from the expected cause high Chi Square values, and high Chi Square values cause low p-values. Low p-values help us understand the probability that A and B are not independent. And this point, about what a p-value means, is quite tortured and debated about. It wasn’t obvious to most real humans until the 1700’s and a human called Pearson didn’t make the connection until 1900. But I’m going to give this knowledge to my Agents for free. (I spoil them!)

chi2, p, dof, expected = stats.chi2_contingency(obs)

I’m going to ask all 20 agents to report on back to me how sure they are that there is no relationship between Attribute A and Action B.

P VALUES
           0
5   0.044430
1   0.069816
16  0.243300
4   0.378896
12  0.399213
2   0.408720
7   0.408720
14  0.460522
19  0.488906
8   0.500817
18  0.545683
0   0.558591
10  0.595316
6   0.669254
11  0.689098
3   0.716851
13  0.776825
9   0.935711
15  0.993611
17  0.993611

Agent 17 reports a 99.3610601185% chance that there is no relationship between Attribute A and Action B. As does Agent 15.

Agent 5 reports a 4.44300910571% chance that there is no relationship between Attribute A and Action B.

And you can see here, the chances that there is no relationship between A and B vary from 4.44% all the way to 99.36%. Now remember, everything in Agent 4’s experience tells them that there are pretty good odds that there is a relationship between A and B.

What effect does giving my agents more sample have? Let’s increase it by a factor of 10.

#set size of the sample to be pulled

sample_size = 1000

And we check out the probabilities that the A isn’t related to B.

P VALUES
           0
6   0.036959
13  0.076433
0   0.096252
7   0.107040
15  0.141339
17  0.161236
1   0.168180
3   0.227854
14  0.229311
4   0.294523
19  0.366309
9   0.368254
10  0.477954
18  0.531879
5   0.546537
16  0.584444
8   0.728194
11  0.874022
2   0.895047
12  0.994750

And we see right away that Agent 6 has something unusual.

Agent
6
Sample Crosstab
A      0    1   All
B                  
0    261  228   489
1    238  273   511
All  499  501  1000

Sample A Mean
0.501
Sample B Mean
0.511
Sample 1-1 Mean
273
CHI STATS FOR THIS AGENT
p
0.0369586132321
chi2
4.35231785869
dof
1
expected
[[244.011 254.989]
 [244.989 256.011]]

Agent 6 observed 273 people with both Attribute A and Action B – so they could predict that somebody doing Action B, given Action, 54.5% of the time, which is better than even odds. They report that there’s just a 3.6958% chance that the relationship between A and B is a fluke. This is their experience. All the other agents report that there’s no relationship between A and B.

What about the ground truth of Universe?

Universal CrossTab
A        0      1     All
B                        
0    25032  24941   49973
1    25202  24825   50027
All  50234  49766  100000
CHI STATS FOR THE UNIVERSE
p
0.36956011438
chi2
0.805143058414
dof
1
expected
[[25103.43682 25130.56318]
 [24869.56318 24896.43682]]

Nope. There isn’t a relationship. There’s no relationship between A and B.

Let’s amp the sample size up to 10,000 and see what happens.

#set size of the sample to be pulled
sample_size = 10000

Agent Zero, even with 10,000 in sample, reported a relationship.

P VALUES
           0
0   0.017687
9   0.061001
10  0.065904
18  0.105137
19  0.122991
3   0.209514
4   0.281474
12  0.326845
13  0.364398
14  0.433251
11  0.519662
1   0.521283
2   0.531240
6   0.560125
15  0.605112
5   0.688567
7   0.890137
17  0.900055
8   0.929782
16  0.955363

Upon closer inspection of Agent Zero, we see why:

Agent
0
Sample Crosstab
A       0     1    All
B                     
0    2524  2505   5029
1    2376  2595   4971
All  4900  5100  10000

Sample A Mean
0.51
Sample B Mean
0.4971
Sample 1-1 Mean
2595
CHI STATS FOR THIS AGENT
p
0.0176866154165
chi2
5.62692654471
dof
1
expected
[[2464.21 2435.79]
 [2564.79 2535.21]]

Even though the Ground Truth, indeed, indicates that A and B are not related.

Universal CrossTab
A        0      1     All
B                        
0    24909  24958   49867
1    25053  25080   50133
All  49962  50038  100000
CHI STATS FOR THE UNIVERSE
p
0.949061729484
chi2
0.00408130392167
dof
1
expected
[[24914.55054 25047.44946]
 [24952.44946 25085.55054]]

Again, Agent Zero isn’t a bad human. It’s just that they got unlucky. Their experience of the Universe leads them to believe that there is a relationship between A and B, even though there isn’t. Even with a sample of 10,000.

In all Four of these Universes, there was no relationship between A and B. Aside from showing you the ground truth and the related statistics, I can prove it by showing you the way I created the Universe.

dict_a = {a: np.random.randint(0,2) for a in range(n)}
dict_b = {a: np.random.randint(0,2) for a in range(n)}

A and B are independent because they were created independently. They were willed into being using two separate random distributions, at two different times (one followed the other) and they have nothing to do with each other, other than existing in the same Simulated Universe together. The fact that it seemed like usually, one agent out of twenty always thought there was a relationship is predictable. It’s just that no agent ever believes that they’d be so unlucky.

At sample = 100:

Sample Mean P VALUE
0    0.419183
Sample Range P VALUE
0    0.933486

At sample = 1000:

Sample Mean P VALUE
0    0.417475
Sample Range P VALUE
0    0.976968

At sample = 10000:

Sample Mean P VALUE
0    0.536264
Sample Range P VALUE
0    0.887396

The P-value range doesn’t become any tighter or more consistent, across the agents, as sample goes up. There is no tightening of certainty at scale.

What if I create a universe where A is related to B?

What if A is correlated to B. What if they’re not only just correlated, but I make A cause B? Naked. Pure. Causality.

dict_a = {a: np.random.choice(np.arange(0, 2), p=[0.50, 0.50]) for a in range(n)}
dict_b = {}

for key, value in dict_a.iteritems():
if value == 0:
dict_b[key] = np.random.choice(np.arange(0, 2), p=[0.80, 0.20])
if value == 1:
dict_b[key] = np.random.choice(np.arange(0, 2), p=[0.20, 0.80])

For each human created from the void, they’ll have a 50:50 chance of getting Attribute A. For those who do not get Attribute A, there is a 20% chance they will do Action B. If they have Attribute A, there is an 80% chance they will do Action B. It is this way because I say it is. The determination of a 0 or a 1 under Action B is determined by, is because of, having Attribute A. It is causal both in terms of time, and in terms of definition. That’s just how I created the Universe. (This is beauty of simulation!)

So I set the agents sample size to 100.

sample_size = 100

And they all come back with very low p-values.

P VALUES
               0
5   7.640166e-13
3   2.970227e-11
10  1.440464e-10
14  5.218926e-10
18  6.594466e-10
8   1.845767e-09
0   1.958641e-09
16  1.958641e-09
13  2.338188e-09
12  2.387777e-09
17  3.596767e-09
11  6.236816e-09
9   1.713752e-08
1   5.138308e-08
4   5.508875e-08
6   5.805862e-08
7   2.820377e-07
2   4.233467e-07
15  4.403962e-06
19  1.220610e-05

And, for intuition, let’s check out why Agent 5 came back with such a low value:

Agent
5
Sample Crosstab
A     0   1  All
B               
0    38   6   44
1     7  49   56
All  45  55  100
Sample A Mean
0.55
Sample B Mean
0.56
Sample 1-1 Mean
49
CHI STATS FOR THIS AGENT
p
7.64016573956e-13
chi2
51.3724911452
dof
1
expected
[[19.8 25.2]
 [24.2 30.8]]

As you can see, one would expect 30.8 people in the sample to have 1′ both A and B, and the observed value was 49. Add up all of these variations, and we get a high chi squared value of 51, which produces a very low p-value. If a person walked into a room with Agent Five and told them they had Attribute A, they’d predict that the person would do Action B 90% of the time.

And that’s within a good range of what Universe says it really is – what the ground truth really is – it’s pretty close.

Universal CrossTab
A        0      1     All
B                        
0    39894   9935   49829
1     9960  40211   50171
All  49854  50146  100000
CHI STATS FOR THE UNIVERSE
p
0.0
chi2
36249.5631917
dof
1
expected
[[24841.74966 25012.25034]
 [24987.25034 25158.74966]]

Alright, let’s boost the sample size to 1,000

sample_size = 1000

And see what happens:

P VALUES
               0
14  2.267577e-89
18  5.482615e-89
6   6.981511e-89
17  7.188154e-89
7   5.035317e-88
5   1.285554e-85
19  7.385426e-85
2   2.342636e-84
15  2.376226e-84
0   2.140627e-81
13  3.776932e-81
10  8.144878e-80
11  4.849325e-79
9   3.542902e-78
16  8.106565e-78
1   1.250593e-75
3   1.762194e-75
12  5.910935e-75
4   6.113513e-75
8   8.353442e-69
Sample Mean P VALUE
0    4.176729e-70
Sample Range P VALUE
0    8.353442e-69

And the Agent’s generally agree with the ground truth of the Universe.

Universal CrossTab
A        0      1     All
B                        
0    40081   9984   50065
1     9951  39984   49935
All  50032  49968  100000
CHI STATS FOR THE UNIVERSE
p
0.0
chi2
36153.7396954
dof
1
expected
[[25048.5208 24983.4792]
 [25016.4792 24951.5208]]

And let’s do it again, with a higher sample for each agent:

P VALUES
      0
0   0.0
17  0.0
16  0.0
15  0.0
14  0.0
13  0.0
12  0.0
11  0.0
10  0.0
9   0.0
8   0.0
7   0.0
6   0.0
5   0.0
4   0.0
3   0.0
2   0.0
1   0.0
18  0.0
19  0.0
Sample Mean P VALUE
0    0.0
Sample Range P VALUE
0    0.0

And these agree with the Universe.

Universal CrossTab
A        0      1     All
B                        
0    39892   9882   49774
1     9978  40248   50226
All  49870  50130  100000
CHI STATS FOR THE UNIVERSE
p
0.0
chi2
36333.9440092
dof
1
expected
[[24822.2938 25047.7062]
 [24951.7062 25178.2938]]

Our Agents are able to make good statements about relationships when God is leaning, 80/20, on the Universe.

What happens to the Agent’s ability to predict if we don’t make it so obvious?

dict_a = {a: np.random.choice(np.arange(0, 2), p=[0.50, 0.50]) for a in range(n)}
dict_b = {}

for key, value in dict_a.iteritems():
if value == 0:
dict_b[key] = np.random.choice(np.arange(0, 2), p=[0.55, 0.45])
if value == 1:
dict_b[key] = np.random.choice(np.arange(0, 2), p=[0.45, 0.55])

In this Universe, there’s a 50/50 chance of having Attribute A, and if given Attribute A, there’s a 55% chance of doing Action A. The dependence of B on A is quite a bit more nuanced, but you see that it’s still there.

P VALUES
           0
16  0.002654
14  0.022877
9   0.028232
1   0.047574
5   0.054960
3   0.101809
10  0.106456
0   0.124586
8   0.133113
11  0.219751
6   0.305201
15  0.355469
2   0.418625
18  0.446463
19  0.525728
12  0.551016
7   0.862065
13  0.902446
4   0.916301
17  0.974134
Sample Mean P VALUE
0    0.354973
Sample Range P VALUE
0    0.97148

Four agents out of twenty, 16, 14, 9, and 1, experienced P-Values that are less than 0.05.

And this Universe is very, very clear – there is dependence.

Universal CrossTab
A        0      1     All
B                        
0    27433  22344   49777
1    22492  27731   50223
All  49925  50075  100000

CHI STATS FOR THE UNIVERSE
p
7.52459087194e-234
chi2
1066.14820878
dof
1
expected
[[24851.16725 25073.83275]
 [24925.83275 25149.16725]]

Let’s see if more data helps my agents make better predictions?

sample_size = 1000

And the P-values narrow:

P VALUES
           0
12  0.000003
5   0.000006
1   0.000012
14  0.000017
11  0.000023
7   0.000148
19  0.000148
8   0.000173
18  0.000192
17  0.000709
15  0.000713
10  0.000942
3   0.002926
9   0.004837
4   0.006401
2   0.008387
6   0.010469
0   0.030132
16  0.057896
13  0.059988
Sample Mean P VALUE
0    0.009206
Sample Range P VALUE
0    0.059985

And in this instance, all agents but two, 16 and 13, experienced P-values less than 0.05. Pretty good.

We boost the sample to 10,000 each.

 sample_size = 10000

And rerun the Universe.

P VALUES
               0
19  2.425200e-33
14  4.308918e-31
15  8.251558e-28
10  8.948502e-28
6   1.468066e-27
13  1.027866e-26
3   1.606491e-26
4   1.693300e-26
12  2.170612e-26
2   3.552389e-26
7   5.560343e-25
1   2.823116e-24
0   1.003131e-23
16  2.671182e-23
18  4.844996e-23
11  1.348288e-21
8   1.514325e-21
17  2.494315e-21
5   5.137547e-21
9   1.729439e-17
Sample Mean P VALUE
0    8.652486e-19
Sample Range P VALUE
0    1.729439e-17

And the P-values narrow even more. All the Agents would agree – B and A are correlated. And they match up with the Universe.

Universal CrossTab
A        0      1     All
B                        
0    27418  22720   50138
1    22104  27758   49862
All  49522  50478  100000
CHI STATS FOR THE UNIVERSE
p
4.27806977209e-235
chi2
1071.87737866
dof
1
expected
[[24829.34036 24692.65964]
 [25308.65964 25169.34036]]

What if I make a Universe where Attribute A is anomalous – just 5% probability that a human would have it, and what if it just gives that 55/45 Action B split? Note that there’s still a causal arrow in the data. It’s just a lot weaker.

dict_a = {a: np.random.choice(np.arange(0, 2), p=[0.95, 0.05]) for a in range(n)}
dict_b = {}

for key, value in dict_a.iteritems():
if value == 0:
dict_b[key] = np.random.choice(np.arange(0, 2), p=[0.55, 0.45])
if value == 1:
dict_b[key] = np.random.choice(np.arange(0, 2), p=[0.45, 0.55])

And I start off the Agents with a sample of just 100.

P VALUES
           0
15  0.187727
2   0.200325
13  0.206507
3   0.289892
1   0.290413
0   0.470615
9   0.472737
7   0.510751
6   0.673700
10  0.727745
16  0.777826
5   0.817648
17  0.817648
11  0.826133
8   0.894095
19  0.905466
18  0.917037
14  0.949945
12  1.000000
4   1.000000
Sample Mean P VALUE
0    0.64681
Sample Range P VALUE
0    0.812273

And we see that none of the agents were able to spot the dependence. Even though in this Universe, it is there.

Universal CrossTab
A        0     1     All
B                       
0    52269  2328   54597
1    42689  2714   45403
All  94958  5042  100000
CHI STATS FOR THE UNIVERSE
p
7.45561589877e-35
chi2
151.676711965
dof
1
expected
[[51844.21926 43113.78074]
 [ 2752.78074  2289.21926]]

So, we’re going to boost the sample to 1000.

sample_size = 1000

And we find that six agents are able to pick it up.

P VALUES
           0
16  0.001622
19  0.007936
3   0.008558
9   0.015707
14  0.018716
13  0.036360
11  0.057232
4   0.089602
12  0.105284
8   0.222948
15  0.244618
1   0.248235
5   0.274044
0   0.327197
17  0.448070
10  0.481531
2   0.513948
18  0.566968
6   0.575433
7   0.602760
Sample Mean P VALUE
0    0.242338
Sample Range P VALUE
0    0.601138

But the Universe says that it’s there,

Universal CrossTab
A        0     1     All
B                       
0    52166  2230   54396
1    42887  2717   45604
All  95053  4947  100000
CHI STATS FOR THE UNIVERSE
p
1.98924420195e-41
chi2
181.771280722
dof
1
expected
[[51705.02988 43347.97012]
 [ 2690.97012  2256.02988]]

In the final run, we give them the full 10,000 sample size

sample_size = 10000

And we find that all the agents pick it up.

P VALUES
               0
14  1.743448e-09
4   1.713640e-08
0   9.902143e-08
7   1.935761e-07
9   7.475067e-07
17  3.601266e-06
15  3.666506e-06
13  5.525651e-06
6   1.075299e-05
12  2.196981e-05
10  2.213179e-05
3   2.672497e-05
1   2.791613e-05
19  4.573572e-05
11  7.288003e-05
16  1.366004e-04
18  1.926298e-04
2   2.279910e-04
8   2.320205e-04
5   2.297855e-03
Sample Mean P VALUE
0    0.000166
Sample Range P VALUE
0    0.002298

Which agrees with the Universe.

Universal CrossTab
A        0     1     All
B                       
0    52402  2256   54658
1    42567  2775   45342
All  94969  5031  100000
CHI STATS FOR THE UNIVERSE
p
1.28720079078e-46
chi2
205.546021449
dof
1
expected
[[51908.15602 43060.84398]
 [ 2749.84398  2281.15602]]

A lot more sample helped the agents understand that a small relationship, within an anomaly (at 5% of the humans having Attribute A) was present in the Universe.

Segmentation

Much of the work that is segmentation is about observing some Attribute A, and running test for correlation against some Action B. The machinery around the word because, is made by the human involved, not the chi square test of the p-value. Chi and p make no statement about causality, they just make a prediction about independence.

The reason for bothering with chi and p-value in the first place is because if one can understand the relationship between an Attribute and an Outcome, they can predict it, and prediction is the key to acting.

Many commercial segmentations one encounters make use of multiple Attributes, like age, gender, income, previous purchase, number of children, marital status, and location, to make a prediction. This increases the number of degrees of freedom, which i turn has a predictable effect on p-values and each agent’s ability to reject the null the hypothesis, or discover exploitable knowledge. In the end, most segmentation boils back to a 2×2 Pearson’s Chi Square test with a single degree of freedom. One either has the attribute of 35+Male+Lower_Income+3children+married+urban, or not. That accumulation of attributes into a single one has the effect of reducing the size of a single cell where the 1 and the 1 line up, that true-positive, sweet spot. One pays the cost of uncertainty by either inflating the degrees of freedom or reducing the size of the segment.

The Problem

The generalized concern, the problem, is the sensitivity of the statistical test for independence, to the size of the segment, is much greater than is generally understood. In the simulations in this post, we demonstrated that even with a sample size of 1000, just six agents were able to pick up (at p < 0.05) on a segment that was clearly there (5% of the population), with an exploitable feature (55:45 for purchase).

In fact, the number of agents that were able to detect the segment were [4, 4, 4, 6, 3] on successive runs.

This took place in a simulation. In a world that I created, with a direct, and I assert, causal link, between A, and then the generation of B. The sample was pulled, at random, from the Universe. And, we knew the Ground Truth of the Universe with absolute certainty (because we created it). There’s none of the usual problems with the real Universe. No census whose credibility we can quibble about. No transcription error. No pesky human with a fat finger. No liars. And nowhere did the God get shifty and changed Confidence Levels randomly. The agents were all super sharp and good professionals.

The segment was there, and, most agents missed it.

If the sample size is increased to 3000, a general size, we get a boost in the predicability – as [12, 13, 12, 10, 15] agents managed to pick it up (p<0.05). Again, that sensitivity, from 1000 to 3000, is the difference between most commercial sampling studies.

Big Problem? Big Data?

So what, one might say, why sample when I have all the data?

We simulated several universes where there was absolute independence between A and B, and we found, consistently, that one agent in twenty was finding evidence of dependence. We saw this often.

At industry conferences, I have encountered practitioners that will argue the correlation must exist if it was discovered, even in the case of correlating an astrological sign with the medication condition of breaking a leg. They insist that it is a problem with the sampling, or that more sample size always solves the problem. However, even with this sterile, simulated, data, sampling 10,000 records out of a ground truth of 100,000, one agent was still tricked by randomness into stating there was an exploitable segment when there very clearly wasn’t one.

You now have more intuition about the structure of the Universe than they do.

Conclusion

Six years ago I wrote WTF repeatedly in the margins during a presentation. What I learned there worried me, and it wasn’t until I started running more simulations on my own that I truly appreciated what was happening deep down there.

The best practical advice that I can offer data scientists in industry is to look for big, bold, segments. Look for big chi squares and very tiny p-values. Be aware that a promising segment might not actually exist even if the p-value is under 0.05, or even under 0.01. You may be an unlucky agent. And you wouldn’t know any different.