Microsoft patents a method to detect toxic content at work

Sinziana Mihalache
by Sinziana Mihalache
Author
0 Comments
Download PDF

  • Microsoft researchers receive a patent for a method that might detect toxic content.
  • If implemented in consumer products, this will definitely improve behavior in work environments.
  • Looking for recommendations on collaboration software? Visit our Teamwork Hub.
  • Find expert advice on the best software to secure your identity at work, in our Privacy section.
microsoft patent toxic content

A few days ago, Microsoft received a patent for a computer-implemented method for detecting toxic electronic content. The focus is on work-related environments.

The method probabilistically determines toxic keywords that indicate toxic content.

It can be difficult to assess and improve the health of a company. Company leaders need visibility into employee engagement, insight to focus on what is most important, and guidance to take effective action. To this end, web-based computing platforms exist to solicit and obtain comments from employees. These platforms allow the company to present prompts for comments to employees in a web-based user interface. Using the web-based user interface, the employees can provide anonymous comments about the company and the employment experience in a free-form text format. […]companies would appreciate solutions that help them more quickly identify toxic comments among potentially tens or hundreds of thousands of comments submitted by employees.

By toxic content, the method refers to comments that are rude, disrespectful, threatening, obscene, insulting, and contain identity-based hate.

If the method will see the light in final products such as Microsoft Teams or related apps, it will greatly contribute to a healthier work environment.

How will a comment/remark be flagged as toxic?

How will a comment/remark be flagged as toxic

As the description of the patent shows, one solution would be to train a deep machine learning classifier to classify comments as to toxicity.

The level of toxicity for a word, comment, or remark will be established given the frequency of use under certain circumstances.

[…]the toxic keywords are determined based on comparing term frequencies of the keywords in a set of example toxic interpersonal electronic communications against term frequencies of the keywords in a set of example non-toxic interpersonal electronic communications. A keyword is determined as indicative of toxic content if its term frequency in the set of toxic examples is more than a threshold number of times more than its term frequency in the set of non-toxic examples.

How often a keyword can appear before it’s considered toxic, is probably a detail that each company should establish.

Although, as the authors of the patent mention, establishing what is toxic or not might consistently vary from one enterprise to another.

At any stage of the process, users can opt in or out of the process. Also, personally identifiable information when using this method may be removed, if desired.

However, while there might still be a long run until such a method is implemented for real, its an important step toward improving work environments.