In this post
The industry default for moderating user content is reactive: publish first, report later, remove if someone complains.
It works at scale, and it fails exactly in the first few hours — which are the hours when content circulates. By the time removal happens, the damage in a serious case is done, and the victim did the work of reporting.
In EuRi, a meme social network, we inverted that. This post is about what the inversion cost.
How it works
user creates a post
-> Cloud Function
-> Gemini analyses
-> passed? yes -> goes live
no -> does notThere is no window between publishing and moderating, because there is no publishing before moderation.
Choosing a Cloud Function rather than doing this in the app is the point that matters most and gets discussed least: client-side moderation is optional moderation. A modified app skips the check and writes straight to the database. The rule has to sit where the client cannot reach, and the only way to guarantee that is for database writes to be permitted solely through the path that runs the check.
What cost more than expected
Latency on the critical path
Calling a model before publishing adds waiting, and the wait is variable.
The real problem is not the time: it is perception. A button that goes quiet for two seconds reads as a freeze. And the obvious answer — a loading spinner — solves it badly, because it explains nothing.
| Approach | Perception |
|---|---|
| Frozen button, no feedback | broken app |
| Generic spinner | slowness |
| "Checking your post", explicit state | a process, with a visible end |
The third also creates the opportunity to explain a rejection when it happens — the difference between a user who understands the rule and one who thinks the app is arbitrary.
False positives
This is where the most time went, and the reason is intrinsic to the content: memes are ambiguous by construction.
Irony, sarcasm, self-deprecation and in-group humour look like aggression to a naive classifier. A keyword filter blocks half of the legitimate material, and a humour app with a filter like that is not a safe humour app — it is a broken one.
Early prompt versions erred on the restrictive side unacceptably often. What fixed it was not tuning severity: it was replacing description with examples.
A prompt that describes offence in the abstract produces an anxious classifier. A prompt with concrete cases — including things that should pass — behaves far better. The list of passing cases matters as much as the blocking one.
AI moderation is not a switch. It is a parameter you calibrate against your own content — and erring on the restrictive side costs you the product.
The failure decision
Any synchronous moderation architecture must answer one question before shipping:
If the analysis service is unavailable, does the post publish or does it hold?
Failing open keeps the app working and opens an unfiltered window precisely during an outage — when nobody is watching.
Failing closed holds content and frustrates legitimate users.
We chose to fail closed, with a visible "under review" state. The reason is specific to this product: if the differentiator is moderation, a period without moderation is not degradation — it is the absence of the thing we promised.
In a product where moderation is incidental, the opposite choice would be reasonable. What matters is that the decision is made deliberately, not discovered during the first incident.
What the inversion actually solves
It removes the circulation window. Serious content does not sit live for twenty minutes.
It takes the burden off the victim. In the reactive model, the system depends on someone being harmed and having the energy to report it. That design outsources moderation work to exactly the wrong person.
It reduces human volume. What reaches manual review is the edge case, not the obvious one.
What it does not solve
Reports still matter. Borderline cases, context and disputes still need people.
Context-only harm. A harmless image aimed at one specific person can be harassment, and no classifier looking at the post in isolation sees that.
Behaviour change. Users learn what passes. A filter that is not revisited becomes a filter that is routed around.
Frequently asked questions
How much latency does this add?
Enough for users to notice. Calling a model on the critical path of publishing is not free, and the response time is not constant. The work was not to eliminate the wait — it was to make it legible in the UI so it does not read as a freeze.
What happens if the model is unavailable?
That is the most important design decision in the system: fail open (publish) or fail closed (hold). We hold and show an 'under review' state, because in an app whose differentiator is moderation, publishing unfiltered during an outage destroys the promise itself.
How do you tune the filter without killing the humour?
With examples, not adjectives. A prompt describing offence in the abstract produces an anxious classifier. A prompt with concrete cases of what passes and what does not — including irony and self-deprecation — behaves far better.
Does this replace human moderation?
No. It reduces the volume reaching a human and removes the window in which the worst content circulates. Reports, review and edge-case judgement still need people.