Bots Ruin the Online Advertising Ecosystem

For professionals in digital marketing, many of your metrics rely on some sort of measurement of views, be it visits, impressions, or reach. Those numbers are supposed to tell you how many people were exposed to your ads. But the majority of web traffic isn’t people; it’s bots. A 2014 report by Incapsula – corroborated by other sources – found that two-thirds of traffic on the internet was not human.

I’ve been speaking with Dr. Augustine Fou, an independant ad fraud researcher with almost 20 years of experience in digital marketing. He’s been trying to determine how to identify and stop bots from affecting digital ads.

When I talk about bots, I’m not talking about those bots that announce themselves as bots like search indexers like Googlebot and Bingbot. Those good bots properly identify themselves so that everyone else can filter them out of results. The other bots claim that they are humans surfing the internet. They try to act as much like a real person in front of a computer or mobile phone as possible, like the replicants in Blade Runner. And they can cause even more damage than those cinematic fakers.

Fake Visitors, Real Harm

Bots are a serious problem for digital marketing. Bot traffic primarily harms two sets of people, publishers and marketers. It’s not just inflating rates. These bots screw up the very metrics that Internet marketing builds off of. “Everyone knows bot traffic and fraud impressions mean wasted ad dollars,” said Dr. Fou. “But even more pernicious is how the bots mess up analytics, causing advertisers to inadvertently send even more money to their evil overlords – the bot masters.”

For advertisers, bots first and foremost cost money. The point of an ad is to get the advertiser’ message in front of a person who might want their goods and services. A bot won’t be buying anything. But the impressions and clickthroughs that they generate cost an advertiser money anyway. Combined with malicious techniques like stacking ads on top of each other and putting them in tiny iframes, an ad fraudster can really rake in the dough. eMarketer estimated that the 2016 marketing spend was $15 billion – a pretty big pie to steal from.

In chasing premium ad revenue, bot fraud cheats publishers out of premium ad traffic and ad dollars. Imagine you have a niche product for sports fans. Through targeted advertising, you could make sure your ad only shows up on sports related sites, like ESPN. If a human visits ESPN, they see your ad and get a tracking cookie. Bots, however, are liars. They can tell the ad servers that their traffic comes from any site they want, as the URL for ad clicks often contains the source of the traffic. And by getting cookies by visiting sites, they can pretend to have whatever interests pay the best.

For marketers, bots wreck their analytics numbers. You’d have no way to know how well a campaign works if over half of the impressions and clickthroughs were due to bots. Your clients want results for their media spend. They get far less for their money when all that spend falls into the hands of bot traffic. And you as the person responsible for creating their ad campaign will look far less effective and valuable.

But middlemen like ad buyers have a pretty big disincentive to cutting out bot traffic. Much of their money comes as a percentage of that media spend. If bots inflate the amount spent on media, they also inflate the amount the marketer gets paid. And imagine how upset that client would be when you tell them their impressions and clickthroughs dropped to less than half in an instant.

Stopping Traffic

It’s not the big sites that cause problems. The long tail of the Internet holds most of the exploitative sites, those that abuse the ad networks. Back in the early days of the Internet, a scammer site could just copy existing content onto their sites, stuff it with ads, and wait for the search traffic to come flowing in. Now, they don’t even have to bother putting up content. All they have to do is flip a switch and send an army of millions of bots to click on their ads. They create the site and the traffic. It’s basically free money.

If this bot traffic is so harmful, why does it still persist? Dr. Fou points out the ugly truth about bot: they make money for the ad networks. “The more flow through, the more profits — for the middlemen.”

So the ad networks aren’t likely to stop bot clickthroughs. You need to do that yourself. But how?

The first method involves detecting bots through technology. Unless you get lucky and the bot tells you that it’s a bot, you need to get a little more clever. Dr. Fou works to find ways to spot a bot. When a user, real or fake, hits a website, it tells that site a fair amount of information about its browser, operating system, and more. Bots may be liars, but they’re not all great at it. Think of it like fake IDs: the ID may look good at first glance, but inspect it closer and maybe you see the hologram is missing or the ID number follows the wrong pattern.

The second means of attack goes after the business motives around ad fraud. Most of their money comes from serving up a ton of impressions really quickly. If you can change from a model that rewards quantity of impressions to rewarding business results, you can start to eliminate those views that don’t produce results. Right now, it’s absolutely worth it for scammers to buy AWS time to run bots to pretend and view ads. “Solving fraud is not hard – if advertisers focused on business outcomes and not on quantity of ad impressions purchased.,” said Dr. Fou. “Then whether or not it was fraud, advertisers would buy less of the stuff that didn’t work – i.e. didn’t drive more business. This automatically reduces ad spend on bots and fraud, because those don’t work anyway.”

Ad fraud is a real problem that eats up a significant piece of marketing budgets around the world. Recognizing how this problem occurs is the first step. After that, you’ve got to figure out how to stop it. In the next article on my talks with Dr. Fou, we’ll reveal various steps you can take to eliminate bot traffic.

All Government Acts are Underwritten by Force

I saw a little bit of this libertarian guy, Dean Smith, talking on the Joe Rogan experience about how taxes as they are now will be seen as a crime akin to slavery by future Americans. He believes that because tax collection is underwritten by the threat of violence. If you do not pay your taxes, he says, men with guns can come and forcibly take you from your home and throw you in a cage. That our government operates using the money taken under the threat of force is criminal.

This misses something important. All governments are underwritten by the threat of force.

Every law, every fine, every regulation, every mandate the United States government issues is backed by men with guns. If you don’t have a fishing permit and catch yourself a monster trout, you’ll catch a fine. If you don’t pay the fine, in some states, you can be imprisoned. The state will attempt to garnish wages or place liens on your assets first, but that is also a form of force.

This is the social contract, and it isn’t voluntary. As a citizen, you cannot opt out except by leaving the country. And then you’ll be subject to the rules and regulations of another government.

But imagine a government where you could opt in. Where you voluntarily pay your share of taxes and, in exchange, receive the services of the government. You call 911 because of a fire or burglary, and the dispatcher has to check their records to see if you’ve paid your taxes. No, sorry, we don’t have a record of that, enjoy your smoldering wreckage.

This voluntary tax scheme would actually build a massive record-keeping bureaucracy to track payments across multiple service providers. If you think the government is flawed and inefficient now, wait until they have to check records every time someone gets stabbed or has a heart attack or crashes their car. How often do you think they’ll get it wrong? Dispatchers already make errors by assuming a call is not an emergency or sending help to the wrong address – they’re only human, after all.

What about things that benefit everyone as a whole, like the military, the CDC and NIH, or the National Weather Service? If you don’t pay your share, do you get the benefits of cancer research? Or do the costs of those studies get passed on to the companies researching cancer treatment, who then pass it on to cancer-riddled consumers? We’re already at a point where prescription drug research is already so expensive for often marginal benefits. Pharmaceutical companies have to pursue money makers, drugs that treat chronic illnesses, like depression and high blood pressure, instead of drugs that cure diseases, like antibiotics. At this point, every single government action becomes a consumer choice. Or, more likely, a corporate investment.

We created governments to handle the big, societal issues that arose from agriculture – defense, food distribution, common good, economic protection – and continued with industrialization – pollution, labor exploitation, increasing globalization. These problems don’t take well to opt-in or consumption taxes because they affect everyone. And while individuals may disagree about the specific large scale problems to solve – look at Lamar Smith’s war on the NSF grant process – I don’t think many people believe that there are large, complex issues we need to tackle as a country.

Taxes are how we fund these solutions. We may have legitimate quarrels over the type of taxes and the rates, but to argue that we shouldn’t have taxes at all is fantasy. Your income depends on the protection and services that the government provides, so you should throw that government a cut. If you don’t, the notorious GOV is gonna come knocking for their money, Lebowski. If there were no consequences for non-payment, there’d be more people ignoring to government. In order to maintain their authority, the government needs to use force to maintain power when people challenge it. That they need to use force show a weakness in their power and authority.

Fortunately, the US maintains a mostly consensus government, where nearly every gets to vote for representatives who determine where their tax dollars are spent. Elections lend tax collection a measure of legitimacy. If you think, taxes are theft, you can vote for people who will oppose them. But you are going to have a hell of a time arguing against the existence of taxation once people realize all the things those tax dollars pay for and the alternatives to mandatory income taxes.

Fiction is an Argument with a Reader

Fiction is an argument about the nature of the world. The author writes their story to persuade you that their insights, observations, and ideas illuminate or reflect the reality that we all share. This applies to even the most fantastical stories, as they have to still convince a reader that the extra-real additions to the story are convincing within a different context (exo-planets with advanced technology) or alternate physics (magic or multiple planes of reality).

Authors persuade through details. The cracks in the ceiling, the inflection and slang, the other riders on a bus, all these work to sell the reality that contains a story. Regardless of where that story takes place, those details have to be as precise and broadly understandable as possible.

I once stopped reading a book because it described a town as being “like any other small Connecticut town.” On top of the rest of the book, this phrase told me that this was a narrow, insular book, one that had no desire to speak forth and reveal. It was the mark of narcissism. The phrase “small Connecticut town” codes a lot into it and will mean different things to different people. That the author used that shows he can’t escape his own shorthand, that he isn’t thinking of other people when he writes.

To be fair, that book suffered the curse of Hemingway, written in a spare and emotionless prose that came off boring instead of efficient. And it covered the dullest and most worn rut in American literature: masculinity. It did not convince me that it had something to say, or, more simply, that the story I was reading was worth continuing.

The simplest fiction stories, those without any assumed changes to the existing world, are the hardest to make convincing. They speak of you and your world. The more you know the setting, the personalities, the conflicts, the sharper the argument has to be. If a story has an old timer drinking his pension away in your neighborhood with can after can of Pabst Blue Ribbon, it won’t be believable if you know those guys order Narragansett because PBR is for broke hipsters, while Narragansett is for broken men.

That sort of persuasion comes from a sort of competent authority, that you have shown that you know this world, that you’ve watched and understood people enough to see them in this environment or extrapolate them into it convincingly. Sometimes that takes a sort of domain knowledge, about the location, about the types of people, or about the actions they do. Other times, all it takes is empathy. Writers (and, hell, anybody) can argue their views best when they a well developed sense of empathy.

There’s other ways to persuade with a story. Because an author defines the world you’ve viewing, they can manipulate it and, therefore, you the reader. All writers manipulate their readers, so watch out about judging that word, manipulate. I’m using to mean any time someone hides some amount of information about their intent from another person. An author can use this to great effect by hiding some information about characters or events, for example, to construct a twist that changes the nature of a story. Mysteries rely on some amount of manipulation to be satisfying.

Manipulation only goes so far for a writer. Persuasion, either through demonstrating knowledge or constructing enough believable details, must work hand in hand. Imaging a murder mystery where the killer is revealed to be a character that had not been introduced yet. You’d feel pretty cheated. Manipulation works only if you hide some facts about a story, not all of them.

Some writers, like John Grisham or Amy Tan, use their personal authority as experts in other domains – lawyers and Asian Americans, respectively – to convince you that the details within their stories reflect reality. I don’t know much about either how lawyers operate what Asian Americans go through, so when these authors throw in words, events, or locations that I don’t understand, I still believe them thanks to their personal background.

But woe betide the writer who does not convince. Their story falls flat, runs listless, and marrs their name. What convinces one person may not convince another, even beyond matters of style and taste, depending on their existing beliefs. Our brains are hardwired to defend our existing beliefs even if they are wrong. Smart people – that is, people more likely to read – defend their beliefs harder. So if you want a good story, cram it with details, accurate and heartfelt, and hope that a few them land with everyone.

Do Women Have it Better in Countries Ran by White People?

A couple of weeks ago, I talked to a random guy at a bar who dropped this gem on me: Name a country that has women’s rights that isn’t run by white people. I was a little flabbergasted, to say the least, that someone who make such an openly racist claim. My first reaction was to ask if he could name a country that had a lot of rainfall but didn’t have trees. Because that felt like the same back-assward way of looking at things.

But the statement ate at me. Why, exactly was this wrong? Was it even true? I know that the US doesn’t have the greatest record on women’s rights – smaller political representation, among the very few countries with absolutely no maternity leave. But even if this statement is literally true, why is it a misguided way of looking at the world?

The first stop is the World Economic Fourm’s Gender Gap report for 2015. This report addresses the gap between men and women’s outcomes in various countries. That means if a developing country has deficiencies in some area – say education or economic opportunity – they only consider the difference between how men and women are treated. If everybody enrolls in the same terrible schools at the same rate, there is no gap. The list ends up being pretty heavily weighted by economic opportunity and political representation, the categories with the biggest variation.

The top three, according to their metrics, are all Scandinavian: Iceland, Norway, and Finland. The bottom three are all Muslim: Pakistan, Syria, and Yemen. Bar guy would have liked this. In fact, 17 of the top 25 are majority white countries. But here’s the thing: these countries all spend a good portion of their GDP on social services. These countries are generally wealthier and spend a lot more of that money through their government. As this chart breaks down, being in a wealthy country that spends money on its people is a better indicator:

Country Overall Gap Rank Political Rep. Rank GDP per Capita (2014 unless noted) Government spending as % of GDP
Iceland 1 1 $52,004.00 24
Sweden 2 3 $58,938.00 25
Finland 3 2 $49,823.00 26
USA 28 72 $54,630.00 15
Pakistan 143 130 $1,316.00 14
Syria 144 87 $2,078.00 (2007) 12
Yemen 145 140 $1,408.00 (2013) 14

Source: World Bank Data from

Having women in politics doesn’t hurt either. I included the USA in there because that was my racist friend’s baseline. As a wealthy country that doesn’t spend a lot through the government, women do okay, but not among the best. And we have terrible political representation.

There’s a couple of non-white countries in the top ten: Rwanda and the Philippines. Neither of these countries is wealthy and neither spends a lot on government. But both have better female political representation and economic opportunity. And both of these countries have about doubled their GDP since 2006.

But this may not have been what he meant. The World Health Organization released a report on violence against women throughout the world. Overall, 1 in 3 women will be victims of violence, either by an intimate partner or sexual violence by a stranger. This graphic breaks it down by region:


Violence against women is generally higher in the Global South, but again, the key indicator is wealth. Places with lower incomes have higher rates of violence against women. If you look at the Western Pacific Region – majority non-white – they have slightly lower rates than Europe. They also have generally higher income than their neighbors to the southeast.

So if women have more rights where incomes are higher, and the places that have higher incomes are generally full of white people, doesn’t that mean that racist guy is still right? Well, no. In fact, it’s not even wrong. His statement has a big ass logical fallacy in it that implies that white people are somehow better as a group because they have the financial security to treat women with some equality. These countries have a sort of Maslow’s hierarchy thing going on: they’ve got a pretty good handle on the biological and safety needs and will help out those who struggle, so that their citizens can think about the top-level items, esteem and self-actualization. Poor countries struggle to feed their people and suffer from preventable diseases like dysentery, developing countries are trying to build out industries, power grids, and transportation infrastructure while battling worker exploitation and pollution, while the rich countries get to think about individual rights and global status.

Hell, the entire idea of civil rights is relatively new. Less than two hundred years ago, you could own people in the US and other supposed enlightened white countries. Three hundred years ago, the major European powers thought nothing of invading places on other continents, claiming dominion, and butchering the populace if they resisted. The US is proof of that. The fact that it is now majority white means that the darker skinned native population that existed beforehand was largely wiped out.

Women’s rights is something we’ve only been working on for about a hundred years. And the European countries didn’t have to deal with freeing themselves from a foreign yoke. Now that they have the ability to make their own decisions (though this is debatable with the demands that any World Bank money come with), give them some time to come around. It took the US about a hundred and fifty years since we kicked our British masters out (not to mention that the people who kicked them out were also basically British). Most of the countries that have less women’s rights have had independence for fifty to one hundred, at best.

Open Carry and the Politics of Personal Power

I’ve been very slowly reading Power: Its Forms, Bases, and Uses by Dennis Wrong (but it feels so right). I say slowly because it’s both a dense academic read and every time I get a few pages in, I start thinking about various forms of power in practice.

I keep coming back to the open carry mindset. By that, I mean gun owners who support openly carrying or wearing holstered firearms in public places. There’s some speculation that openly carrying a gun deters crime – criminals see armed citizens and decide against robbing that 7/11. Speculation, but little conclusive proof. To me, that’s the closest I get to agreeing with the open carry mindset. It’s like having a bunch of uniformed officers walking around a troubled neighborhood. The power of the visible.

But open carry is more about expressing the power of the individual carrying. It’s a show of potential force that indicates the guy with the gun has power. They never need to act on that show – in fact, it’s a sign that their power is weak if an individual or entity has to resort to using force. But the display itself is their source of power.


Most of the open carry supporters have a strong mistrust of the government in all forms, preferring the smaller, tribal unit of the family. And the gun is there to protect the family. To show that this family unit is strong and can defend itself. The comedian, Bill Burr, talks about the standard paranoid situation where someone breaks into your house with the sole intention to kill you and your family. But that’s just paranoia. In 2011, there were about 100 homicides associated with burglaries. A small fraction of the total deaths in a year.

This and other justifications used for openly carrying guns – the imminent and constant threat of crime, a tyrannical government oppressing you – are fears that a malicious actor will impose their will on you by force. To counteract that potential force, they need to have access to force themselves.

When you are afraid of being acted upon, you feel powerless. The gun is their access to power. As Mao Zedong said, “Political power grows out of the barrel of a gun.” It applies to all forms of power, though, as having access to force, you can compel others to do your will. Or, in this case, prevent others from forcing you to do their will. It’s a strangely logical mindset when you burrow into it.


The difference between myself and the open carry movement is that I don’t believe there is a reason to arm myself in public. I would rather support better policing, as I believe in the promise of government. The far right sees government as a dangerous other, drunk on power and ready to take advantage of us. I see government as us, the collective expression of the population’s will. But maybe we’re both projecting on government, I with my hope and trust of fellow man and the paranoid right’s fear of being taken advantage of, placing on it those qualities we guard like diamonds in our hearts.


You Are Not Weird

I went Google fishing the other day using the other day using the phrase, “I have a weird job.” What I wanted was people confessing to sanding dildos and training spiders. What I got was people with pretty normal jobs who posted things that were a little silly and had no context.

For example, Austin journalist Emily DePrang posted this image of her inbox subject lines:

These are mildly amusing. One of them uses the word ‘abortion’ in a casual, jokey way, which, OMG, edgy. Way to freak out the squares. The other makes a pun using a well-known 40-year-old song. Nobody has ever done that.

This was the top result and it pains me how boring it is. Other not weird job results include a radio DJ posting the amusing, out of context names of sound clips; Jonah Goldberg misunderstanding the concept; somebody building what I assume is a thermin (my sound is off because courtesy); Jennifer Lawrence; a guy who updates a website that sells electric bikes with really specific guidelines; some guy from Fox Sports who thinks Hulk hands are about the weirdest thing ever; and plenty more mild to unusual situations in pretty mundane jobs.

I did find one legitimately weird job. A brief video where someone rubs paint on another person’s belly and they rub around on a piece of butcher paper. Whether the person is the rubber or the rubbee, that’s weird to have that as a job.

The thing is, people aren’t that weird. Being in a job where you have to come up with entertainment is not that weird. Writing amusing subject lines with co-workers or having a phone call about a giant squid being treated like a rare leopard for pretending really well is not that weird. There may be weird situations and things that, when shown out of context, seem absurd. But you aren’t weird.

A lot of the people who claim weirdness in these situations are actually super normal, boring people. I hate to break it to you. These are high achieving professionals who have colored within the lines in order to achieve a profession that is stimulating and has some variety. People who are legit weird usually don’t get that kind of notice unless it’s negative. But the concept of being “weird” or “crazy” has some cultural cachet, the idea of the mad genius, who taps into some sort of person ley line to draw out an unseen truth, who lives life truer than the rest of us.

I see with the stand-up specials on Netflix. Anytime someone uses the word “crazy” or “weird” in the title of their special, I think, “Man, this guy is gonna be boring.” The two that come to mind are Jeff Dunham’s “Spark of Insanity” and Donald Glover’s “Weirdo.” There’s also one called “Completely Normal” by a guy named Tom Segura, but I have no idea what he’s like, so I won’t talk shit about him. But the other guys, whoa. Jeff Dunham is probably the biggest offender. He does a ventriloquist act for the blue collar comedy tour set. Pretty straight down home stereotype stuff about cranky old people and terrorists. But he does it with his hand up a puppet, so whoa, watch out world. Donald Glover, who I like as a person, is just sort of boring on there. It seems like a guy with a showbiz career trying to branch out.

Painfully Normal.
Painfully Normal.

I get miffed about people claiming weird, because I know some weirdos. I know a guy who once obsessed about the Fibonacci series so much that he incorporated it into doodles and chord progressions. I know a guy who has a semi-mystical philosophy that’s equal parts quantum physics and Aleister Crowley. I’ve met folks who make their livings as dominants and submissives, but without any sex. I know a guy who makes ambient noise music and when I met him, offered me a few drops of his custom tincture of mood altering herbs and supplements. These are proper weird people and none of them would ever say that they themselves are weird.

People who are actually weird don’t advertise it. They don’t have to, because they don’t care if they are weird. They just do the things that interest them, public image be damned. We all can learn a lot from our local weirdos.


Why Isn’t My Doctor in My ACA Network?

When I quit my job last May, I also quit my health insurance plan. Not on purpose, really. Like every other health plan I’ve been on during my adult life, it was provided and mostly paid for through my employer. But now that I am footloose and fancy freelance, I’ve got to pay for my own insurance. So I got one through the Affordable Care Act (ACA) exchanges.

I paid a little extra to get a plan that had both my primary care physician and the ENT who has been making my sinuses slightly less terrible. But when I got my insurance card, I had a different physician listed. At the ENT office, they seemed surprised that they were even listed there. “We’re not in any of the ACA networks,” they said. “We’re already in so many networks that it’s not worth it.”

I liked those doctors a lot. But none of them are in any ACA networks. And they aren’t the only ones. According to a study by Avalere Health, plans offered through the ACA exchanges have 34% less providers than the average employer-based or individual non-exchange plan. So you may be able to keep your doctor if you like them, you just have to pay full price for services.

According a report by the Robert Wood Johnson Foundation, 40% of ACA plans are considered small or extra small – that means that less than 25% of available providers are within their networks. I’m lucky; I live in New York City, so I can find another primary care provider nearby. But can you imagine the huge hassle it must be if you live in a small town? 25% of doctors could be one.

Insurance companies are narrowing networks in order to lower the rates they charge. They get pickier about the doctors they use – they want a high quality to efficiency ratio. On the face of it, that seems pretty good, right? You get solid, waste-free doctors, sawbones who get the job done without cost overruns.

But this isn’t exactly right. When the insurance companies narrow their networks to lower costs, it’s not just because they are being choosy. The costs go down because they reimburse less for procedures. Doctors will join these narrow networks because the limited number of available doctors means that more patients will be sent to each individual practice. Narrow networks create volume-based health care providers.

So why are some doctors not on any ACA networks at all?

“The exchanges have become very much like Medicaid,” says Andrew Kleinman, president of the Medical Society of the State of New York . “Physicians who are in solo practices have to be careful to not take too many patients reimbursed at lower rates or they’re not going to be in business very long.”

The short answer is that doctors who take a lot of ACA patients like me will go broke. Now that we get to see the actual costs of insurance instead of having secretly tacked on to the end of our pay as a benefit, we want cheaper plans. But that means lower reimbursements to doctors (the whole reimbursement dance is clusterfuck for another post), which means less doctors sign up. Klienman says the reimbursement rates can be less than 50% of what the commercial plans pay. Plus there the huge deductible you have to burn through before insurance kicks in a pays anything.

On top of that, I have a 90-day grace period for my insurance payments, but the insurer won’t pay reimbursements unless my account is paid up. Doctors on the network have to take me, but won’t get paid until I remember to write that check.

We’re creating a two-tiered system for insurance. People like people who don’t get insurance through their job have less access to providers. On top of that, we get access to doctors who are willing to take lower rates for more patients – either doctors without enough patients, those willing to run a volume business, or new doctors. Single payer, where for art thou?

I still thought I was getting what I wanted. But no, the directory was wrong. And that’s a growing problem. Brian Hoyt, managing director at Berkeley Research Group, wrote in a recent white paper that “Provider directory inaccuracies represent a growing and significant risk both to consumers and health plans. Inaccurate directory information may limit a consumer’s ability to verify if a preferred doctor is in-network, or to know how many and what types of providers would have to be accessed under a particular product offering.”

These inaccuracies have led to lawsuits against the insurance providers. Which is going to raise my health insurance premiums.

I know this is the beginning of a new system. Under the old system, I would have gone for the cheapest plan possible, which was no insurance at all. This way, at least I’m covered if I get hit by a car and they find a brain tumor. But I’m less likely to consider getting that checkup to catch the tumor early.