How to evaluate an AI assistant built into your membership platform
Every membership platform has an AI feature now. Almost none of the sales pages say what it does with your member data, and that is the only question whose answer changes your risk.
You do not need to understand how language models work to evaluate one. You need five answers, and a vendor who cannot give you all five in writing has told you something useful.
The five questions
1. What member data can it see?
The answers range from "the record you have open" to "everything in your account". Both can be reasonable. What matters is that the vendor can state the boundary precisely, because a vendor who describes it vaguely has usually not drawn one.
Ask specifically whether it can see financial records, member notes, and anything staff wrote assuming no one else would read it. Membership databases accumulate candid remarks about members. Those notes were written for an audience of two.
2. Does any of it leave the vendor's systems?
Most platforms do not run their own models. They send data to a provider such as OpenAI, Anthropic or Google. That is normal and not in itself a problem, but it means your member data is crossing a boundary and you should know which one, and whether it crosses in identifiable form.
The follow-up question is the important one: is the data used to train the provider's models? Business tiers usually say no. Get it in writing rather than in a demo.
3. Can it act, or only draft?
This is the largest single difference between products, and it is where the risk actually lives.
An assistant that drafts a renewal email and waits for a person to send it can be wrong at no cost. An assistant that sends the email, applies a payment, changes a member's status or issues a refund can be wrong expensively and without anyone noticing until reconciliation.
Ask which actions it can take without a human clicking approve. If the answer includes anything that moves money or changes a member's standing, ask how you turn that off.
4. Can you switch it off and keep the rest?
An AI feature that is load-bearing is not a feature, it is a dependency. If disabling it breaks search, or reporting, or the renewal workflow, then you have not been offered a choice.
The test is simple: ask whether an organization that turns the AI off entirely still gets the product they are paying for. The answer should be yes without qualification.
5. Do members get told?
If AI drafted the email a member receives, or scored their engagement, or summarized their history for a staff member, there is a disclosure question and in some cases a privacy one. Under PIPEDA, members have a right to know how their personal information is used. Provincial rules add to that, and Quebec's Law 25 goes further on automated decision-making.
You do not need a vendor to have solved this. You need them to have thought about it, and to tell you what is disclosed and when.
Four things that should stop the conversation
The AI is a rules engine with a new label. Some products describe conditional logic that has shipped for a decade as artificial intelligence. Ask what it does that a filter could not. If the answer is a list of thresholds and triggers, it is a filter. That is fine, and it should not be priced as AI.
No isolation between customers. In a multi-tenant platform, every organization's data sits in shared infrastructure. Ask how the AI is prevented from surfacing one organization's data to another. A vendor who has not built that boundary has not thought about the failure that would end their business.
It can send or charge without approval. Covered above, and worth repeating because it is the one that produces an incident rather than an annoyance.
There is no opt-out for a member. Some members will not want their information processed this way, and a few will ask. If the platform has no mechanism to exclude a member, the organization has to answer that request manually or not at all.
What good looks like
A useful AI feature in this category has four properties.
It is grounded. Answers come from your own records and cite which record they came from. An assistant that produces a plausible number with no source is worse than no assistant, because it is confidently wrong in a format that looks authoritative.
It is additive. Turn it off and the platform still does everything you bought it for. The AI removes work rather than being the way work is done.
It stops before money and status. Drafting, summarizing, finding and suggesting are all safe. Sending, charging, refunding and changing a member's standing wait for a person.
It is specific about the boundary. The vendor can tell you which data goes where, in what form, and what the provider is contractually allowed to do with it.
How we handle it
Ask Sembr answers questions about your own data and shows which records it used. It drafts, it does not send. It never changes an invoice, a payment or a member's status.
Member names and documents are pseudonymized before anything reaches the model, so the provider receives the shape of the question without the identities in it. The platform is fully usable with AI switched off, which is not a setting we added late. It is an architectural rule we wrote down before building the feature, because an AI capability that becomes load-bearing takes the choice away from the customer.
Use the five questions on us too
The framework above is not a Sembr checklist dressed as advice. It is the list we would want a chamber to run against any vendor, including this one.
Send the five questions to every platform on your shortlist and ask for written answers. The differences in what comes back, and in how quickly, will tell you more than any demo.
Our answers are on the AI page, and the data-handling detail is in the subprocessors list.