Ask HN: Handling customers that want feature previewing?

[cbono1]

Curious how you folks manage the release process with larger customers that ask for advance notice (ex: 2 weeks) or previewing of any features before release or general availability?

We have a single sandbox environment currently, but different customers are asking for varying amounts of delay before feature release. How do you manage bug fix deploys in these cases as well?

[Arubis]

Most of the discussion here will be technical, especially around feature flags and friends. That’s fine, but please don’t miss out that this is a clear signal to charge these customers a lot more money.

[talolard]

I'll offer an illustration. We sell our SaaS at X. We have a customer that needed stronger SLAs and servers located in Europe. They pay 8X and (I think) both sides feel like they got a great deal.

So to paraphrase Arubis, this is a business question,not a tech question. At what price point is the technical investment covered ?

[ambivalents]

> a clear signal to charge these customers a lot more money.

Interesting, can you explain this a little more? Some of our customers chomp at the bit for particular features, and when we can we give them early access. We do not charge them more for arguably huge amounts of value add to the product.

I wonder how to have these types of conversation, internally and with users. I'm a PM but have a direct line to management. I'm frustrated with how much money I believe we are leaving on the table, even if it doesn't directly benefit me.

[Macha]

Feature flags is usually how we handle this. We have the flags anyway for developer use, so making them user specific was not that much incremental effort though we have had a few cases where old workflows were not removed when they should have been.

We did once have a customer with their own environment with a different version deployed, as they were initially a major client with such demands but after a few years it became clear it wasn't worth the maintenence effort to special case this one client so we made it clear in contract renewal one year that this would be going away.

[notJim]

+1 on feature flags. Set up your flags so you can disable features for certain customers, or groups/classes of customers. Delivery and release can then be separated. Release frequently, deliver features when it makes sense.

Also recommend making sure customers know that this is time-limited, otherwise you end up supporting too many different configurations of your product.

Also if these are customers you're in regular contact with, you can preview your roadmap or demo features that are coming soon. This helps them feel in the loop, and gives them a chance to provide feedback.

[williamdclt]

That's about the only answer, although feature flags are hell. Once you have a feature flag, you now have 2 versions of your product. 2 feature flags, that's 4 versions to maintain, etc. Make sure you don't conflate "feature flag" and "setting", and make sure you clean up these flags when they're uneeded (and actively push the business to make them unneeded!).

Fowler has a good write-up about different types of feature flags: https://martinfowler.com/articles/feature-toggles.html

[petters]

Agree on this. We delivered new versions to customers once a week, but every new version should not have been noticed by a customer at all.

The roll out of feature flags is a separate process.

[tombot]

+1 for flags, you can also put features into production earlier, safe knowing they will only be visible for the users / teams you’ve added to the list

[1a527dd5]

+1 on feature flags. I find them to be a pain, but fussy customers insist "that we need to test it in staging first".

[geocrasher]

Sounds to me like the customer is trying to solve a problem and is coming to you with their chosen solution. My suggestion: Find out what problem they are trying to solve, and address that.

Also, you might look at cPanel's notices that they send out where they explain upcoming feature updates well in advance of the rollout. They also have different release cycles, where the newest features are released to the EDGE cycle instead of the STABLE, and then the long term release cycle remains unchanged except or security updates.

[mooreds]

> Sounds to me like the customer is trying to solve a problem and is coming to you with their chosen solution. My suggestion: Find out what problem they are trying to solve, and address that.

This.

Problems they might be trying to solve:

   * want to be able to jump on new features as soon as they are released, but need to plan with the eng team
   * want to be "in the know" so not surprised by new features
   * want to be able to give feedback and help direct the roadmap
   * have been bitten by bugs in your software (or other vendors) and want burn in time, especially if you upgrade them without their choosing (as is typical in some SaaS models)

I'm sure there are others, but it's worth asking why they are requesting this if you haven't done so already.

[plasma]

Have you clarified why they want this notice?

In my experience it’s because if you roll out a new feature (or non-trivial UX change), that can wreck someone’s day and cause internal support issues into the company that uses your product.

Consider your rollout of new features to be least impacting, don’t confuse someone who’s using your product to get work done by changing something that breaks a workflow that worked yesterday etc.

You can put new features in new sections, or need special users to activate them.

In some cases we would show screenshots of upcoming features to hear about any other feedback, but otherwise focused on new features not being disruptive.

[cbono1]

Thanks for the reply - yea these are great points to consider and raised a few of them. Ideally rollouts benefit all customers, and the concern is that there is some possibility that something might not jive with them and they want the ability to do testing ahead of time.

Seems like putting as many things that might risk this under settings/configurable as possible? This does extend development to have to gate everything in this way though.

[dsr_]

Context: financial SaaS of interest to banks, brokerages and RIAs; no direct-to-consumer at this time.

We have two special case preview environments. They're set up just like production, but with lower capacity and no SLA.

The first one is general previews of what we're planning on releasing; it's a normal part of our release process and every release is there for 2-14 days before going to production. All customers have access; some use it extensively, and some basically ignore it.

The second one is for specific in-development features; in theory we can spin up multiples of this, but in practice we don't want to diverge too wildly. Generally we showcase one feature at a time, and we gate the features by customers.

[theoblank]

I would seriously consider not allowing special treatment in cases like this. In my experience things like this can be a slippery slope. Decide what is most broadly reasonable for both your company and your customers, communicate it well, and then spend time and energy on your core product.

[Jugurtha]

Figure out why they're asking that. This appears to be an "XY problem"[0].

An example of this is a friend asking me how to weld a wire to a thin metallic plate. I digged further to find the actual problem for which his solution was to weld the two. He said it was a fuse's wire that melted and he wanted to weld a bigger wire to avoid that, to which I replied: don't. The whole point of the fuse is sacrifice to protect down stream equipment.

The same when a relative asks how to format a disk as their solution to solve a problem.

Generally speaking, implementing customers' solutions is a recipe for disaster, especially with enterprise.

Ask what it is they are really trying to accomplish, and figure out a way to solve that. Listening for problems instead of solutions or implementations.

- [0]: https://en.wikipedia.org/wiki/XY_problem

[browningstreet]

I had a similar situation in fintech. We had 3 different release cycles and hosting farms for fast, medium and slow accepting customers. Fast went to the customers who didn't require approvals, and who were flex when it came to errors in the release. Then there was a medium channel where the other customers were fine as long as a couple of the major customers in that pack approved things. The slow channel basically was for customers who had large validation or training requirements associated with releases.

[cbono1]

Would this break down if you had 10 customers that would prefer the slow channel of release, but each had their own ever so slightly different desired release pace or training requirements?

[cweagans]

"Thank you for your inquiry. This is not a service that we offer at this time -- our testing and release process is fully automated and there is not a way for us to exclude some customers and not others.

Can you please help us understand the purpose of the advance notice/release delay is? What benefit does a process like that provide to you or your organization?"

---

I generally use some variation of that ^^. Having some customers on different release tracks is very painful in my experience, especially if it's done without properly planning for it in your release process. For example, let's say that right now, anything committed to `main` goes out to production via some automated process. What happens when you have a security bug that needs dealt with ASAP? In my org, this usually means that the security patch gets rolled out to everyone, and then when it's time for customers on a slower release schedule to "catch up", there are conflicts to resolve manually.

It's just not worth it. Figure out why they're asking, but say no. If it's a problem with buggy releases, that's the _actual_ problem to put some time into solving.

[johncessna]

The argument I always got while trying to argue against delaying features for large customers was that it causes a training headache. If you have large enough customers, redesigning a UI triggers a training initiative for the 100s, if not 1000s of customers that use your software.

[ghiculescu]

Lots of good answers below on how to make it work, how you should charge more, etc.

Remember that saying no is also an option. You aren’t obliged to say yes just because someone asks for something you don’t sell, even if they are willing to pay.

For most SaaS there’s no logical reason why having a feature for 2 extra weeks would be an enormous advantage to a customer. In my experience what this question really is is “can we make last minute change requests to a feature you thought was done”. I’m not keen to guarantee that, so it’s a simple no, sorry, we can’t do that. It’s yet to be a deal breaker.

[cosmie]

> For most SaaS there’s no logical reason why having a feature for 2 extra weeks would be an enormous advantage to a customer. In my experience what this question really is is “can we make last minute change requests to a feature you thought was done”.

If you frame it as early access, then people are apt to try to throw their wishlist at you, thinking it's still "not official yet" and easy to change.

But based on what OP asked, I'm not sure that's the primary reason. It seems more like his larger clients are finding the changes disruptive enough that they're wanting advanced notice to prepare for them. So integrating a lot of the suggestions here can very easily address that issue, while also providing leverage points for client management and sales.

@OP - don't think about it in terms of preview access or letting them into your internal sandbox environment(s). But rather, restructure your releases to decouple the release from the account migration. Using feature flags and such then allow end users to determine whether they immediately migrate to the new feature when it's available, or if they continue with the status quo and wait a few weeks to familiarize themselves with the changes before opting into them.

For end users – this gives them room to both continue with their day-to-day needs using the process they're already familiar with, while giving them flexibility to familiarize themselves with the changes and prepare (in whatever form they need) before committing to them.

For the tech team – there's an initial lift to adjust the app architecture to support this type of usage, but insulates them these sorts of client requests in the future since the app now supports these types of opt-in migrations within the live environment. If a user wants a sandbox account, it can be provisioned as a normal account in the live environment but with sandbox-like restrictions applied to it.

For the sales/account teams – this can be used as a negotiation point. You can position it as normal accounts get auto-migrated to new features upon release, but enterprise clients can pay for the ability to control their own release/migration timing via these opt-in feature gates. And a sandbox functionality (which provisions a new account using the current configuration of their existing account, but with applied sandbox-like limitations) allows them to both prepare for it and test it in a controlled environment before enabling it on their primary/production account. That sandbox capability is independent of the opt-in capability, and a second point of negotiation.

And since your application now supports both of these directly in production and doesn't require bespoke work or custom environments from your tech team to support, the incremental cost is pretty minimum to give it to a particular client. Which is hugely useful for a sales team when closing larger accounts. They can give it away without much hassle if they have to in order to close a deal or provide value-add during renewal, while simultaneously capturing incremental revenue from clients that are willing to pay substantially for these sorts of things.

[bkuehl]

There is a lot of talk of feature flags but that only works when you can realistically delay your feature 2 weeks so that the 'special' set of clients can see it early. What we've seen that has worked is essentially demoing the feature in staging/test to stakeholders at the major/demanding client. They are happy that they aren't taken by surprise and then we provide training materials for major feature/UI changes to their users. Never have had an issue, client is happy and essentially zero technical involvement.

[pkrotich]

We use release branches (git) for such cases - mainly because our deployment is based-off release branches, thus gives us flexibility to move a customer to a pre-release or even have then stay at a particular release forever. Bug fixes can be back ported as needed.

Tried feature flags but it introduced complexity.

I wouldn’t charge more as others are suggesting- bugs you’ll address before general availability is worth the trouble.

[cosmodisk]

I really like how Salesforce do it: they make an announcement that a feature/fix will be available from day x and depending on its importance, they then give 100,200,500 days or whatever to install it or it gets installed automatically on the deadline day. This gives plenty of notice to customers to prepare( updates can break some customisation on client's instance).

[zerkten]

You need to consider the blast radius and how long features may be in preview. Ideally, you would scope the preview to just a subset of users. When you go to a complete customer base you start to get into the situation where the customer has failed to communicate with their users, and have more people raising issues. This isn't bad necessarily, but you have to be prepared to handle it.

If you keep something in preview for a long time then there can be a divergence from the main product. Make sure that data or work from users is migrated across when you finally release.

It makes sense to think about the opportunity for support to participate in the preview. Let them listen in to customer calls and learn about the problems. Providing a dedicated support person for previews is tempting, but long-term it's better to avoid this specialization and enable the whole support team to participate. This shares knowledge and avoids certain people being granted special privileges while others toil on average support tickets.

Microsoft publish some of their methodology at https://docs.microsoft.com/en-us/microsoft-365/admin/manage/.... The key point is that they have set out the boundaries for their program so that customers know what to expect and it can be repeated for every customer.

You will have to learn how to do your previews, but you should lay down some rules. Set it up so kudos from participants can be used for marketing after. You are the one to decide when the preview ends, but setting some minimum time in preview gives customers more confidence and time for you to process feedback. Make it clear if features are locked when you enter the preview and what feedback you are after.

Once the preview is over, thank the participants. Call them out if you have a public forum. Participation can become a mini case study for your marketing efforts.

[leesalminen]

I also had a handful of customers like this. We ended up creating an opt-in “beta” channel that we treated as a first-class citizen along with “prod”.

Basically we had 2 production stacks with CI/CD for each. The only shared thing was the AWS ALB. Our customers each have their own subdomain so we used that to point to different EC2 ASGs.

We would continue pushing bug fixes to “prod”. Once a beta period ended, we merged to master and re-deployed master to both prod and beta channels. Rinse and repeat.

It works out pretty well. Occasionally there’s a really important bug fix that we have to cherry pick from prod to beta, but it’s rare enough to not be a huge concern.

Oh, and, charge them for it. I never got any pushback.

[cbono1]

Thanks for sharing!

Was the beta channel that you had in the pipeline for prod? i.e. prod would have no chance of getting ahead of beta because prod was always deployed based on what was last on the beta release?

Hot or bug fixing is an interesting topic because you could end up with diverging histories if you're not careful. Did you try to maintain `master` as the single source of truth or did you make separate release and deploy branches for each environment?

[jpmoral]

Do the two stacks each have their own database? Have you had customers wanting to move back and forth between channels?

[codegeek]

Charge them extra for a "staging" environment setup only for them and then deploy all latest stuff there for them to test. Then get their approval to apply it to their production environment. Charge well for this.

[cbono1]

Would you have the staging environment share data with production or have its own infrastructure? or use feature gating? One customer may not care that it shares the same DB, but another might.

How would this work at scale? say if there were say 10 customers wanting approval, you'd have to have separate code release branches/versions for each customer? I imagine this could get error prone.

I suppose to your point on charging additional fees for this you can afford any sort of workflow.

[tboyd47]

You need at least 2 sandbox environments: one that's more development-oriented (little to no restrictions on merging), and one that's more release-oriented (as close as possible to production).

Ideally you have 3, and the third one is not used except to demo new features to customers AFTER they have been released (and perhaps, hidden behind a feature flag).

If you only have one sandbox, then it becomes a brawl between development, sales, and management.

More than three is overdoing it and assuming a technical burden for no benefit. (No, you don't need the capacity to spin up sandboxes willy-nilly. Three is enough.)

[winkelwagen]

Difficult to answer without understanding how you are deploying currently. I do think feature flags can be an anti pattern. I would ask the customers why they want it in the first place. Is it a trust issue? Is it an internal process that they require? Ideally you would want to stay flexible, that would mean not tying your company to their schedule. It is a hard application to deploy multiple times for each customer. Perhaps a more lts approach for those customers might work. Just keep in mind the larger the releases get, the more issues it might cause.

[jitendrac]

I would suggest pre-defined staged rollout based on customer preference. For example, create three category Alpha, Beta, Stable.

Alpha: Gets new features immediately and gets support within 24hours if something breaks.

Beta: Gets new features Every 45 week, and gets priority support if needed.

Stable: Gets new features quarterly, Mostly they are forced to upgrade from old features which you dont want to support, But they will be essential volume of your revenue.

alternatively, you can give them sandboxed demo read-only(resets every night at 12 o'clock) account for testing.

[OriPekelman]

well, self horn here... but that is a basic functionality of https://platform.sh (CPO here). You get per branch full copies of production in isolated staging environments that are very cheap to run. So you just clone production to a specific branch apply whatever changes and grant access to the clone. ..you do need to run production with us for the magic though... and a PaaS is not for all use-cases (specifically when running a SaaS...).

[villasv]

We pick those customers to be beta testers. They receive pre-release features, but in return they must accept possible bugs will happen (of course, when one is identified we give them the choice of disabling the feature immediately).

The schedule is totally in our hands, though. Once we say it’s time to go GA, it goes. At this point there still are feature toggles, though, just in case.

[Hamuko]

If a large customer wants to decide when they get new features, they need to pay for their own dedicated environment that is on a separate release cycle. Otherwise they need to adapt to our release schedules and make sure they get all the previewing that they need on a test environment.

[antonpirker]

We create a separate branch for the customer and preview this branch with wunderpreview and give the customer the link to the preview.

Every once in a while we merge new stuff in the customers branch so she has always the latest stuff in her preview.

[quickthrower2]

We have a powershell script that:

1. Copies the production do 2. Removes sensitive data 3. Deploys the chosen build of that app (from Appveyor) pointing to that db

So in a nutshell CI/CD pipeline

If you are having trouble perhaps there are a lot of manual steps to your deployment.

[runwaykid]

One way is to make use of AppConfig to store customerwise feature roll out plan. To save cost, add 15 minute caching on it (the thick client comes with the local application caching feature)

[mbrodersen]

The solution is to ask those customers to pay for the extra work required. If they refuse then it isn’t important enough for them.

[that_guy_iain]

Seems like you need advanced feature flags.

[cbono1]

In your comment here, what's the distinction of advanced vs. a regular feature flag?

[Animats]

Is it that the customers want early access to new features, or they don't want you breaking the product?

[bri3d]

Once you know you need to do this (and at every enterprise SaaS company I've seen or worked at, you eventually need to do this - the sibling comments about charging more $$$ are smart though):

* "Release train ahead" - code takes 2 weeks (or 4 weeks) to reach Production from merge, but it goes out to a Preview environment immediately or at the next release. So, you've forked your code into a Prod train, a Preview train, and mainline. You probably need the ability to hotfix to prod and preview fix to the week-ahead-of-prod preview environment, as well as roll normal development forward. You do gain "soak time" for code in the Preview environment, which is a slight advantage. And your QA is easier, because you have three discrete, testable things ("does it work in Prod, does it work in Preview, does it work in trunk"). But, you have another runtime environment which can diverge, which is a big disadvantage.

* Feature flagging by "release train," single execution environment. This is probably the most "Software Theorist Friendly" way to do this. You don't have to maintain/debug multiple environments beyond what you're already doing. Some customers are in the "preview ring" and get more features toggled on. Single runtime environment, single codeline. Fixes are just fixes. QA is a little harder depending on how you set up the possible "rings" and combinations, and the operational blast radius of your Preview code is now the same as your Production code (make a bad query that kills a backend service in Preview code, well, that's also Prod code. Oof.).

* Free-range feature flagging. Some complex arbiter system manages a huge range of feature toggles on a per-tenant/per-customer basis. Having this range of operational knobs is very fun and works great for customers who are particularly risk averse. It also lets you sell SKU-packs of flag combinations, roll specific features to specific "Early Access Programs," and so on. But, you get a combinatorial explosion of feature flag permutations, which depending on how coupled your code is and how intelligent your QA system is, can become a disaster: "Oh no, FooFeature only works with BarFeature on but our QA system always has BarFeature on so we didn't catch the regression! And only one customer has BarFeature disabled for legacy reasons!"

* Some combination of 1, 2, and 3 - for example a Preview environment that runs the same _code_ as Prod, but with the toggles set differently. A few feature flags which can be enabled for early access programs, but a general rule that code needs to live in a release train. This is my favorite approach. Having a separate Preview environment eliminates the blast radius concern of Preview-toggled code breaking Prod. Operational costs are higher, but these can be passed to the customer (making customers buy preview environments as a line-item is pretty popular). But by maintaining single codeline (rather than deploying a fork of last week's code to Prod), the code-theory is still easy, you don't have divergent release trains, and you don't have the combinatorial explosion that comes from a free-range feature flagging system.