Sentiment 101 - FAQs from our Customers
Just wanted to push some information out there addressing some of the common questions we get here @lymbix. We love talking about this kind of stuff, so any more questions don't hesitate to fire them at us! (or comment)
"We’ve checked out Sentiment and it sucks…"
Thats often what people say in the space, mostly because there is a misunderstanding with respect to the capabilities of sentiment. First off, there's no "right" way to measure it. There are some different schools of thought when it comes to how to directly measure the effectiveness of sentiment analysis, but there has yet to be a standard. Mostly because of our human subjectivity that comes into affect. You feel differently about something than I do (just the way that we see things differently). A simple example of this would be to consider talking about a new pen that you bought the other day to a friend; "Man, this pen is comfortable but its not good to write with". Here we have 2 different sentiments expressed in one simple statement. So then asking, if you were were to tag this as positive, negative, or neutral.. what would it be? Eventually if you ask enough people you'll find that the statement is too ambiguous to determine whether it is in fact a positive or negative statement. That's where we come in; we've created the first ever engine that can measure the levels of subjectivity that could arrise in a given statement, to through off the "accurate" judgement of the sentiment. Of course, its helpful to still know which is mostly likely the "best guess" of the sentiment, but when it comes down to actually using this, you might consider it to be less impactful than a statement that is a lot more clear. Lymbix calls this "clarity".
"Do you do sentiment in other languages?"
Because roughly 60% of north america speaks english (96% US), we jumped on board to developing our lexicons to that language first. The beauty of our system is the fact that we developed a methodology to understand language and sentiment, so we're very loosely coupled to english; meaning of course that it would be easy for us to focus our learning efforts on another language, such as spanish, german, french etc,. The only factor in this is cost, and what we've been doing is looking for the right partner to specify which language we should cover next. That is to say, once we've established a positive working relationship with english, we'd be happy to jump into another language once we've identified that with this partner that it will be profitable right away. Safer for us to proceed this way, rather than gamble on which language will be the next "right" choice for us.
"How accurate is your sentiment technology..really?"
We love this question, because it's the one that lets us talk a bit about how other people(sentiment companies) are gauging their "accuracy". Let's first start by making a determination between what is article versus aggregate sentiment. Nine times out of ten, if you hear a public statement regarding the accuracy of a sentiment engine, it will be on an aggregate level. Aggregate means you take all of the individual articles in a given dataset and determine the overall sentiment or trend of the data. This is helpful, and is usually highly "accurate" (roughly 90%). Thats because the error is hidden. And when I say error, I'm really referring to the misnomers in the sentiment processing. Error is inherent, its always there, and we have to accept it. It's an imperfect technology, based on imperfect data, review by an imperfect process. Then again, does 10% really potential error really make a difference in what value you're taking form consuming millions of articles and making determinations on the results? Now looking a bit deeper, lets consider article sentiment. This is any sentiment engines worst enemy, and thats because now we have to deal with you, the imperfect (but incredibly competent) reader. This really refers to the question 1, the subjectivity in how you feel when you read something compared to someone else. On average, you'll see "accuracy" scores around 50-60% with a good sentiment engine. Always look at the fine print, or impose the question when trying to measure the "quality" of sentiment results. So how does that pertain to Lymbix? Well, on aggregate, we're 90-95% - and thats validated by our customers who use our aggregate measurements. On an article level, which by the way, we absolutely love, we're variable based on your levels of need. If you want 90%, guess what, we can achieve that! but it takes some work. We're a learning network, meaning that we get better with more feedback (our automated feedback mechanisms is a whole other conversation). First off, we start assuming that 50% is bad (not that it is, but we push our engine as hard as we can). So we call that, the inherent 50-50 recall. Now, on the 50% of data thats "good", or "acceptable", we don't measure "accuracy". We measure, "agreeability". That is to say, based on our clarity measurements, how much does our readers "agree" with our results. We've done massive tests and studies to indicate that our article results idle around 75-85%... not bad.. now how do you get better? Well, thats for our paying customer when we turn on our learning algorithms specifically for your datasets.
