Ed Tech User Engagement Challenges and Solutions

These are some challenges that I've encountered while doing user research within EdTech and some of the ways we solved them.

At my time at Discovery Education I’ve encountered a number of challenges while doing certain types of research within Ed Tech, though I’m sure this applies in many other industries I can only speak to what I know. I’ve addressed these challenges, with the help of some amazing peers along the way, in a number of different ways and I would like to share not only how we did it, but what our circumstances were, and hopefully communicate what we did, and why we did it. For the most part the focus is on direct user engagement and research. Some of the methods we use are interviews, usability testing, prototype feedback and testing,

Challenge: Increased Security

It’s fairly common for teachers, and almost always with students, to use a school provided computer and as a result, tend to be restricted in what can be installed. This presents a number of conflicts with tools that we use to facilitate remote user research. For a number of years most of our interviews were done by phone without the ability to interact, assist, or observe users. Often software is required for video calls and couldn’t be installed by the users. This resulted in increased technical difficulties and data being compromised at a higher rate than we liked. Furthermore, it really hindered the ability to connect with users and develop a relationship. Facilitating any type of session with more than one person was made more difficult by the pure fact that no one could see each other and participation would often cause users anxiety. This was discovered as we talked to users over time. Moderation was also made more challenging since there wasn’t any users control when on phone lines.

Solution: Internet based chat programs

We leveraged tools like appear.in that didn’t require as elaborate of a permissions model as other services. Eventually programs like Ring Central offered hybrid experiences were we could still use the regular tools and reduce our library of solutions. The challenge that still remains and won’t ever change is that screen sharing almost always requires the users manually installing something, although I found a creative way around this in some regards I’ll speak to later on.

Solution: Mailing cameras to people

We actually ordered action cameras and built custom stands for users to use and then mailed them with pre-paid postage to return. When mailed back, which usually took a bit of time, we would then digest everything. This method was time and financially intensive and really only implemented when we had something imperative.

Solution: Figma

We leveraged Figma’s prototype and multiplayer modes to get around installation and security issues. This enabled us to follow and record the exact mouse actions, interactions with prototypes, and offer better support and facilitation of research.

Challenge: Distractions and Unplanned Participant

You sit down for an interview and suddenly there are three people instead of one. Now we can’t run a true usability test and naturally this will have some element of discussion to it. We need to pivot on the type of moderation and plans in order to facilitate on goal findings while making sure to retain a positive relationship and dynamic with our users. It’s not limited to EdTech, but I’ve noticed it more within the classroom due to certain schools and district administration being more hands on then others.

The degree that this affects your study is relative to what you’re trying to do. It’s a significant bump when you’re trying to run a usability test or if you’re planning on conducting only a few sessions.

Solution: Have a plan, be explicit, and be flexible

There’s not much you can do about this one besides being pro-active, diligent, and flexible. We’ll often lean more heavily on the planned open-ended questions we usually leave for discussion at the end in this case. Having a fallback is imperative as there isn’t really an elegant way to tell someone that they’re not invited.

Challenge: Choosing the Appropriate Content

I’ve done research with countless different user groups throughout my career and, in my experience, teachers are the hardest time breaking their mental models in order to participate in a research session.

What I mean by this is that teachers generally have a scope of grades and subjects they align do and deal with daily. When testing, teachers generally don’t even think about the content, it’s just assumed it’ll be the same content they’ve been interacting with. It’s when the math history teacher finally sees the content for the session is Language Arts based, and gets thrown off. Even when within the same domain, often if it’s off by some margin, there’s a bit of friction you end up having to overcome.

We don’t often have production level access to user information when exploring and developing solutions, so work done in early phases of project development aren’t able to facilitate verbatim user profiles and scenarios. We could recruit users with specific subjects, but if we’re not testing content, then that just limits our population of users to choose from and makes it all the more challenging.

Solution: Duplicate instances of prototypes with custom content

One of the solutions we tried was to cater to each teacher with broad enough content that it could make sense to them more easily. While we did see benefits and positive changes, the pure amount of additional effort didn’t really merit doing this very often unless at a high volume of users.

Solution: Frame the content and scenario prior to the session

This isn’t unique, but just emphasized more due to the fact that teachers tend to have a specific scope of curriculum that they cover. Often times the content is delivered differently as a result of different subjects being taught in different ways. Simply put, content is not orthogonal layout in our use cases.

Challenge: Limited Availability and Access

Generally asking for anyone's times throughout the day is a big ask. Everyone has their own life, and in the case of teachers. they have the lives of their students as well. I don't have to go over how overbooked the modern teacher is, but that doesn't stop them. Teachers show a much higher sigh-up rate per invite than any other user base I've ever dealt with. They're also the most self-less and excitable users and have no problem doing interviews off-hours in the early morning or into the late evening. I've seen teachers book time when they were well aware they couldn't make it because of how excited they were to participate. Which led us into rethinking the time availability of remote user research and leveraging tools to accommodate orderly scheduling.

Solution: Calendly aided by being remote

With Calendly we were able to open up time slots for entire days in order to accommodate for the busy day of teachers and each other as was arranged logistics. By automating this process, it saved time and stress resulting in being less hesitant to act on doing studies.