Not just software developers. I was working on a medical device and visited doctors' offices to see how and where they stored their medication (e.g. for how could we make our dispenser more prominent without getting in the way?). We also filmed our investigators holding various prototype devices to see how they picked them up, how they were balanced, how they felt in small and large hands etc.
Universally we were told that nobody had ever asked them anything like that before.
I've never understood how people can design and build anything (software, hardware, whatever) without understanding how it's used. Dogfooding is great (and so many places don't even do that!) but you don't really know what you need to build until you sit down with your users and actually watch them work.
Well, I've never been called to participate in a political survey poll thing, but that doesn't mean it doesn't happen to people. There's a lot of doctor's offices out there. What was your sample size?
It was a bit of a shock for me to figure out that much of my early success in software boiled down to a few things I was doing that nobody else was, and one of those was watching other people work and doing something - anything - more with that information than simply writing it down or complaining about it. Most of those people have been coworkers, not even customers.
Agreed - I credit my entire career to this. I started watching people trying to solve problems, and then would piece together something "silly" that helped them solve it.
Next thing I was doing a big system implementation. Then consulting.
One note - now that I'm removed from day to day and don't have time to program myself I see how easy it is NOT to pay attention to how anyone will use something.
I imagine this is how windows gets bloated with candy crush saga and all the bloatware even on the "pro" versions. What employer wants this stuff on users computers? It just clutters up menues.
I don't doubt that this kind of invasive data collection can help developers at some point. But as a user I'm glad that most of the software I'm using doesn't track everything I do (or even register me as a distinct user at all). I'm also willing to accept poorer software quality in return (which curiously, isn't the case).
Having worked on user data before, I think the issue is that for complex software, it’s hard to get value/insight from this data. Imagine if you are Microsoft and had every action every Excel user makes with office365. Where would you even start to be able to get value out of this data?
I worked on the design of the first several versions of Excel. For Excel 3.0 we were intensely interested in user behavior. We often compared alternative prototypes of proposed features in the usability lab. We sometimes would invite develpers to watch users strugging with what they thought was great idea...
Also, long before the Internet we asked users permission to send them a special version of Excel (called the Instrumented Version) that would intelligently log actions, but not data to a floppy disk. Users would send the disks back to us for analysis. We used the analysis to understand how users really did things vs what we thought the did. Sometimes we got some surprises. We used this to prioritize features where users were having difficulty that needed special attention. It also indicated frequenty performed actions that would benefit from, for example, being put on the toolbar.
I think our user focus was an important reason why Excel has been so enduring. (Although Office 2003 screwed it up to some extent - I was no longer there.)
Back then at Microsoft the applications group tended to take the lead in UI developement and many of the things we designed made their way into Windows. Some were even picked up by the Mac! Although Steve Jobs would never have admitted it.
It grieves me to see the shoddy state of software design these days. There is no craftsmanship. Visual design take precedence over real usability. Maybe this is because with fast developemnt cycles and the ability to instantly change a web based design, it's easer to throw crap at the wall and keep iterating until it sort of works.
These days user interaction designers have endless great user data on which to craft wonderful UI but it rarely happens. If they happen to hit on something good, usually another team takes over and changes it, often for the worse!
> Where would you even start to be able to get value out of this data?
It's an interesting question, but I think there are a few obvious easy places to get value (UX on common operations, most common error modes/corrections, etc.)
In my experience the problem is more that devs apply logging or telemetry where it is easy rather than where it is insightful. So if you didn't design the system up front to capture user behavior with some granularity you end up with a bunch of questionable proxies for it, which are hard/impossible to interpret.
I propose you start by studying a small batch of users, documenting their behaviors, develop a set of theories, then look at whether the larger pool of data agrees with or disproves your theory.
Try to come up with theories just from data and you end up with the Texas Sharpshooter scenario. We are complaining about this sort of behavior showing up in the scientific community, let's learn from that mistake.
Install an accelerometer in the keyboard. When heavy impacts are detected from someone smashing their face into it repeatedly, pop up a dialog box asking them if there is something wrong and would they like assistance? Extensive feedback will be forthcoming.
A bit more helpfully, MS were asked repeatedly not to fuck with the menus but they replaced it with the ribbon. Ditto that terrible new interface that was nothing like windows 7. They could have allowed the choice of new interface but they did not. MS got plenty of info and chose to discard it.
MS is now in a position where it doesn't need to care.
> Where would you even start to be able to get value out of this data?
Much like excel for certain professions, I work on a piece of software that is used all day by people in another profession.
One thing we do is look at which features are used most (and least). We make it easier to find and use the most popular features and we make rarely used features less prominent (and in some rare cases we might remove them altogether).
In our most recent redesign, we elevated some of the most commonly used information to the point where the user doesn't even have to click anything to gain some high level insight (previous versions needed at least 1-2 clicks to get to that same info).
We have about 30 power users who we work with continuously. Some of them are nearby, and we often go look over their shoulder and pick their brains. Some of them are in other regions where we get valuable global feedback that we wouldn't otherwise get from those regions.
They get early access to new features and they get to help shape the product that they rely on for their livelihood and help us decide on what we should be working on next. It's win/win and I would recommend forming this kind of relationship with a subset of your user base if you can.
>Where would you even start to be able to get value out of this data?
I'd start with the most undo-ne operations. Which specific actions are most often immediately undone by users (that could lead to addition of functions, changing UI, adding visibility of the effects of an action, all sorts).
Then probably look at most searched terms in both Excel help and online, segmented by user experience if possible (to allow improvements for noobs and advanced users, whilst aiming not to be deleterious to either).
I'm not in software development though, curious what answers y'all have?
FWIW as a relatively novice LibreOffice user I probably most often undo auto-fills where a reference variable hasn't altered how I hoped. (I'd maybe fix that with a tailored tooltip - probably actually implemented as an info box - fired on long-hover of the auto-fill handle).
I read about Office 2000's dynamically hidden menu items feature (with the expand button added to the bottom of each sub-menu) somewhere, that the items that _are_ shown initially were determined by collected usage data. So it's probably aggregation that makes the most sense, not digging into each individual user's behavior.
Well, mcDonalds draws maps of where their workers put their feet, and then designs the kitchens to minimize the number of steps the workers needs to take to complete their jobs.
One way to use Excel telemetry could surely be to minimize the number of cognitive steps workers needs to take to complete the tasks they are trying to do.
Another could be to use telemetry to see at which point a smooth flow suddenly takes a longer time - and perhaps split an action into two actions which are faster (in time) to accomplish
I'd say just invite me over for a chat. I've probably spent 10,000 hours in Excel and can tell you more about its shortcomings than a PB of telemetry ever will
You can do so much with this data, I'm surprised if it's not already collected. You can stratify your users based on what kind of commands they do, how their mouse or cursor is moved around the screen, and what functions they are getting out of excel, and see if you can improve their experiences by lowering unneeded friction and building new features.
I thought about adding something about privacy, but that's sort of a different topic for a different time. There are plenty of privacy preserving ways of learning from this data.
Those same companies mostly load their software and websites full of trackers against all reason. They are very very happy to collect that data. They just never use it - management personalities usually override any data findings... if anyone even ever looks at the results.
I like some of the information in this, and unfortunately I missed the conversation when this made it to the front page a couple weeks ago, I just wasn't in the right headspace.
But I disagree with one of the value judgements:
> As you add more and more users, you learn less and less because you will keep seeing the same things again and again.
No! The frequency of errors is your priority list for UX fixes. You may not, as they claim, find more errors with more than 15 people, but more data samples gives you information about distribution and frequency of errors (do people make this mistake once in a while or every single time?).
You might not need any of that data to make a rough priority list, or it might be helpful for negotiating scope. It'll depend a lot on context.
I take some comfort in the fact that as more and more data gets collected about me, Big Sibling will be drowning in so much data that they can't possibly make sense of it all. Eventually an nearly infinite amount of analysis will conclude that I am boring actually as boring as was initially estimated. That said, I'd be pleased with less monitoring of myself.
Reminds me of this talk[1] I've seen a while ago where an ex-NSA executive talks about how the NSA has too much data to handle it efficiently (amongst other things).
I'm working on an app and trying to be conscientious about my user's behavior, but it's proving to be extremely time consuming for the value I get out.
The problem for me is I can take a relatively accurate guess and achieve 90%+ of what the user is looking for. That last 10% takes a ton of time to figure out. For every moment I spend on the 10%, I'm only getting a fractional return compared to simply moving onto another feature.
Maybe in the future, I'll come back and figure things out, but for now it's just not worth my time.
With a few lines of code, you can integrate Segment (or a similar tool) into your software. You can start tracking user behavior and events automatically. Segment can then send this data onto tools like Mixpanel or Amplitude which has built-in user activity flow analytics.
Take a look into some of these tools if you are looking for an easier way to track user behavior.
I would love to know which features of my app are used and by how many user. But this clashes with my requirement to not have any server side service or telemetry.
You can use a tool like Segment to track user activity client side. With a few lines of code in your app / front end you can be sending user behavior and events to Segment via their API. No custom server side service is required.
Might be worth digging into if you are interested in better user activity tracking.
[+] [-] gumby|6 years ago|reply
Universally we were told that nobody had ever asked them anything like that before.
[+] [-] taneq|6 years ago|reply
[+] [-] squirrelicus|6 years ago|reply
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] hinkley|6 years ago|reply
It was a bit of a shock for me to figure out that much of my early success in software boiled down to a few things I was doing that nobody else was, and one of those was watching other people work and doing something - anything - more with that information than simply writing it down or complaining about it. Most of those people have been coworkers, not even customers.
[+] [-] privateSFacct|6 years ago|reply
Next thing I was doing a big system implementation. Then consulting.
One note - now that I'm removed from day to day and don't have time to program myself I see how easy it is NOT to pay attention to how anyone will use something.
I imagine this is how windows gets bloated with candy crush saga and all the bloatware even on the "pro" versions. What employer wants this stuff on users computers? It just clutters up menues.
[+] [-] t0astbread|6 years ago|reply
[+] [-] ABeeSea|6 years ago|reply
[+] [-] gxx|6 years ago|reply
Also, long before the Internet we asked users permission to send them a special version of Excel (called the Instrumented Version) that would intelligently log actions, but not data to a floppy disk. Users would send the disks back to us for analysis. We used the analysis to understand how users really did things vs what we thought the did. Sometimes we got some surprises. We used this to prioritize features where users were having difficulty that needed special attention. It also indicated frequenty performed actions that would benefit from, for example, being put on the toolbar.
I think our user focus was an important reason why Excel has been so enduring. (Although Office 2003 screwed it up to some extent - I was no longer there.)
Back then at Microsoft the applications group tended to take the lead in UI developement and many of the things we designed made their way into Windows. Some were even picked up by the Mac! Although Steve Jobs would never have admitted it.
It grieves me to see the shoddy state of software design these days. There is no craftsmanship. Visual design take precedence over real usability. Maybe this is because with fast developemnt cycles and the ability to instantly change a web based design, it's easer to throw crap at the wall and keep iterating until it sort of works.
These days user interaction designers have endless great user data on which to craft wonderful UI but it rarely happens. If they happen to hit on something good, usually another team takes over and changes it, often for the worse!
(End Rant)
[+] [-] ska|6 years ago|reply
It's an interesting question, but I think there are a few obvious easy places to get value (UX on common operations, most common error modes/corrections, etc.)
In my experience the problem is more that devs apply logging or telemetry where it is easy rather than where it is insightful. So if you didn't design the system up front to capture user behavior with some granularity you end up with a bunch of questionable proxies for it, which are hard/impossible to interpret.
[+] [-] hinkley|6 years ago|reply
Try to come up with theories just from data and you end up with the Texas Sharpshooter scenario. We are complaining about this sort of behavior showing up in the scientific community, let's learn from that mistake.
[+] [-] tempguy9999|6 years ago|reply
A bit more helpfully, MS were asked repeatedly not to fuck with the menus but they replaced it with the ribbon. Ditto that terrible new interface that was nothing like windows 7. They could have allowed the choice of new interface but they did not. MS got plenty of info and chose to discard it.
MS is now in a position where it doesn't need to care.
[+] [-] esotericn|6 years ago|reply
You know when someone is like, writing 1 then 2 then 3 then 4 then ... or whatever? Yeah, Clippy 2.0.
"Did you know this hotkey called X exists"?
Multiply that by N users - bam you've saved the human race thousands of hours. I'll take my Nobel now.
[+] [-] bonestamp2|6 years ago|reply
Much like excel for certain professions, I work on a piece of software that is used all day by people in another profession.
One thing we do is look at which features are used most (and least). We make it easier to find and use the most popular features and we make rarely used features less prominent (and in some rare cases we might remove them altogether).
In our most recent redesign, we elevated some of the most commonly used information to the point where the user doesn't even have to click anything to gain some high level insight (previous versions needed at least 1-2 clicks to get to that same info).
We have about 30 power users who we work with continuously. Some of them are nearby, and we often go look over their shoulder and pick their brains. Some of them are in other regions where we get valuable global feedback that we wouldn't otherwise get from those regions.
They get early access to new features and they get to help shape the product that they rely on for their livelihood and help us decide on what we should be working on next. It's win/win and I would recommend forming this kind of relationship with a subset of your user base if you can.
[+] [-] pbhjpbhj|6 years ago|reply
I'd start with the most undo-ne operations. Which specific actions are most often immediately undone by users (that could lead to addition of functions, changing UI, adding visibility of the effects of an action, all sorts).
Then probably look at most searched terms in both Excel help and online, segmented by user experience if possible (to allow improvements for noobs and advanced users, whilst aiming not to be deleterious to either).
I'm not in software development though, curious what answers y'all have?
FWIW as a relatively novice LibreOffice user I probably most often undo auto-fills where a reference variable hasn't altered how I hoped. (I'd maybe fix that with a tailored tooltip - probably actually implemented as an info box - fired on long-hover of the auto-fill handle).
[+] [-] rzzzt|6 years ago|reply
[+] [-] draugadrotten|6 years ago|reply
One way to use Excel telemetry could surely be to minimize the number of cognitive steps workers needs to take to complete the tasks they are trying to do.
Another could be to use telemetry to see at which point a smooth flow suddenly takes a longer time - and perhaps split an action into two actions which are faster (in time) to accomplish
I could go on all day.
[+] [-] airstrike|6 years ago|reply
[+] [-] asdff|6 years ago|reply
[+] [-] tinus_hn|6 years ago|reply
[+] [-] bgdnyxbjx|6 years ago|reply
[+] [-] skybrian|6 years ago|reply
[+] [-] mrbbk|6 years ago|reply
[+] [-] izacus|6 years ago|reply
[+] [-] pictur|6 years ago|reply
[+] [-] hinkley|6 years ago|reply
But I disagree with one of the value judgements:
> As you add more and more users, you learn less and less because you will keep seeing the same things again and again.
No! The frequency of errors is your priority list for UX fixes. You may not, as they claim, find more errors with more than 15 people, but more data samples gives you information about distribution and frequency of errors (do people make this mistake once in a while or every single time?).
You might not need any of that data to make a rough priority list, or it might be helpful for negotiating scope. It'll depend a lot on context.
[+] [-] petepete|6 years ago|reply
[+] [-] briandoll|6 years ago|reply
[+] [-] jakub_jo|6 years ago|reply
[+] [-] PopeDotNinja|6 years ago|reply
[+] [-] t0astbread|6 years ago|reply
[1] https://media.ccc.de/v/SHA2017-402-how_the_nsa_tracks_you
[+] [-] squirrelicus|6 years ago|reply
[+] [-] SkyPuncher|6 years ago|reply
The problem for me is I can take a relatively accurate guess and achieve 90%+ of what the user is looking for. That last 10% takes a ton of time to figure out. For every moment I spend on the 10%, I'm only getting a fractional return compared to simply moving onto another feature.
Maybe in the future, I'll come back and figure things out, but for now it's just not worth my time.
[+] [-] sjlogan|6 years ago|reply
Take a look into some of these tools if you are looking for an easier way to track user behavior.
[+] [-] yoz-y|6 years ago|reply
[+] [-] sjlogan|6 years ago|reply
Might be worth digging into if you are interested in better user activity tracking.
[+] [-] JohnFen|6 years ago|reply
We don't need more of this.
[+] [-] iamaelephant|6 years ago|reply