Mastering Survey Questions for AI Comment Analysis

Organizations worldwide constantly seek ways to understand, grow and retain their employees. This can be done through surveys, although many organizations limit the open-ended questions they ask or don’t ask them because text is challenging to analyze. This leaves an abundance of knowledge on the table – knowledge that can be used to prevent attrition or fix broken systemic processes. 

Through its unique text analysis capabilities, Explorance MLY equips organizations with the capability to gain specific insights into the employee experience, ranging from their thoughts on the organization’s strategic direction to how they feel about career advancement opportunities. It is the only solution on the market catered to Human Resources leaders trained to analyze data in terms they understand. 

This article will showcase what makes Explorance MLY unique and how to write questions optimized for artificial intelligence (AI) comment analysis. 

How Does Comment Analysis with Explorance MLY Work? 

MLY is trained on real employee comment data that is reviewed and tagged by a dedicated team of Explorance annotators. Reviewing a comment, the annotation team decides what topics are within a comment and the sentiment behind those topics. The team uses a special type of machine learning called supervised learning, which first requires unstructured data as free-flowing text.

 The unstructured data is turned into “structured data” by categorizing or labeling. The resulting categorized or structured data is inputted into an algorithm. With enough examples in structured data, the algorithm learns the correlation between label/category and what is in the text and ultimately learns to create structured data from unstructured data.  

Applying Supervised Learning to Unstructured Data

The Explorance annotation team receives unstructured data through comments from feedback surveys from our clients. Comments are read one by one and manually categorized by applying all appropriate labels, a process called “annotation.” Let’s consider the following potential employee satisfaction survey comment: 

”My manager is kind and knows how to do our job. He is always willing to lend a hand.”

A data annotator would choose amongst a set number of topics which labels are appropriate for the above comment. In this case, they might decide to tag the comment with the following labels:

“Caring>Direct Manager>Positive; Management/Leadership>Direct Manager>Positive; Support>Direct Manager>Positive.

This annotated comment is then sent to the algorithm. By feeding annotated comments into the algorithm, it learns the correlation between text and labels by example. In this case, the algorithm learns that when it sees a statement like “my manager is kind,” the appropriate label is “Caring>Direct Manager>Positive.” It then transfers this knowledge to tagging texts on its own. 

Challenges with Annotation

One major challenge the annotation team faces is the “poverty of the data.” The “poverty” refers to a lack of information in a comment needed for the model to categorize the comment. The more ambiguous a comment is, the more difficult it is for the model to predict a response. The input must contain sufficient detail and context to encourage the model to categorize accurately to choose between categories.

If a comment says “Flexibility,” the model would have difficulty making an accurate prediction. Is the comment referring to scheduling flexibility? Flexibility as a quality of a person? 

The more details and context in the comments, the more information the model can predict accurately. As it were, the fly in the ointment is how commenters answer questions or prompts. The nature of responses is that they often come in the form of incomplete sentences, point forms, or lists and leave out important details or context. Responses of this type can make it harder to annotate accurately. 

Asking the Right Questions for Better Comment Analysis

A commenter’s natural tendency is to give as few details as possible when answering surveys. Therefore, questions should be written to encourage the opposite. In other words, questions should be written in such a way as to encourage answers that are clear and have details and context. The best data quality for analysis is comments written in full sentences that give as much detail as possible. 

Don’t be too leading

Open-ended questions are good because they encourage the commenter to provide more details. However, questions are often written to contain lots of information, which can encourage shorter responses with fewer details. Since the question already presents the context, the commenter will feel less inclined to repeat details. 

Adding open-ended questions to a survey is often a conundrum. We want the specific details to understand better whatever we might be asking about on our quantitative questions, but we don’t want to burden our workforce with needing to write a lot. Furthermore, without a tool like Explorance MLY, there’s no way to analyze the data effectively.

The power of MLY lies in its breadth – the ability to extract many topics from a sentence or two, assign sentiment to those topics, identify recommendations, and flag alarming content. Thus, it’s essential not to be too leading with your question. With Explorance MLY, you only need a single question to get all the insights. However, it’s important to ask the right type of question.

Our advice: Give direction, but only a little direction. Consider this question: “What do you believe are the company’s strengths regarding employee satisfaction?” This opens the door to a wide array of responses – some might speak about the culture, the growth opportunities, or the people. This is ideal for understanding the full spectrum of what matters to people, and it can be done in a single question instead of 3. 

Ask for examples and details

Besides asking broader questions, don’t forget to nudge your workforce to be thoughtful in their responses. Encourage them to give examples and detail; this can significantly enhance the depth of the data you collect. It can provide context and nuance to their answers, so instead of just knowing the ‘what,’ you’ll understand the ‘why’ and ‘how.’ This extra layer of information can lead to more comprehensive insights into their experiences, opinions, and behaviors.

Detailed examples can also guide decision-making. With Explorance MLY, real-world examples are pulled out as direct recommendations or highlight areas of concern that might not have been on your radar.

Complete sentences rather than short answers

It’s also important to ask respondents to write in complete sentences. Complete sentences provide context and a full picture of their thoughts. This helps understand the reasoning behind their answers, which might need to be recovered in shorter responses. People communicate in complete sentences in everyday life. Asking for the same in surveys lets respondents answer in a manner that’s natural to them.

Get started with these three questions below: 

1. How would you describe the overall work environment at our company? 
2. How would you describe the work-life balance at our company, and what could be done to improve it? 
3. What specific changes or improvements would you like to see in our company culture? 

Questions should be asked individually

Each individual question or prompt should have a single response field. When there are multiple questions with a single response field, it can be difficult for the data annotator, and by extension, the AI model, to interpret which part of the response corresponds to which part of the prompt. In such cases, getting an accurate or detailed response from the model can be challenging, even though we have the question or prompt. Therefore, multiple questions with a single response field are problematic as they encourage this type of ambiguity. Ideally, the question needs to be written so that the link between the question or prompt and the response is clear and unambiguous. 

Also, questions that refer to other questions are tricky because some of the contexts can get left behind in the previous question/prompt and its response. 

For example:

Question: “If you answered “negative” to the last question, why not?”

Response: “The pay was really bad.” 

This type of data would need to be clarified for the model. The question here refers to the answer to a different question, but by reading this question alone, we don’t understand the full context. The model doesn’t consider other questions and responses when categorizing a given comment. It interprets each question-answer pair separately, and therefore, due to the question format, there is information being lost. It is, therefore, essential to write each question so that it can be interpreted with full context in isolation.  

In conclusion

By mastering the art of crafting questions, you can prompt your respondents to share their thoughts in their own words providing a goldmine of unfiltered insights. By striking the right balance between clarity and detail, you lay the foundation for AI tools like Explorance MLY to extract deeper insights from respondents’ narratives. With skillful question design, you empower AI to translate human expressions into actionable data for informed decision-making.