Pulls Tweets with same text and different id's

[austinpgraham]

I found that the issue in the labeler pulling the same tweet is that our database contains tweets with the same text but different id's, sometimes upwards of 20-30 times.

[cegme]

Can you post some examples? Also post the code that is used to generate the tweets.

[austinpgraham]

For example, this query:

Query: SELECT t1.status, t2.status FROM tweets t1, tweets t2 WHERE t1.tweetid = 849076591657353217 AND t2.tweetid = 849076590088736770

The result:
(u'RT @Datosdeunamor: M\xc9DICO JAPON\xc9S REVELA COMO MATAR DE RA\xcdZ LA BACTERIA DE HELICOBACTER PYLORI QUE PROVOCA GASTRITIS, \xdaLCERAS Y M\xc1S!\u2026 ',

u'RT @Datosdeunamor: M\xc9DICO JAPON\xc9S REVELA COMO MATAR DE RA\xcdZ LA BACTERIA DE HELICOBACTER PYLORI QUE PROVOCA GASTRITIS, \xdaLCERAS Y M\xc1S!\u2026 ')

In my test data set (364 tweets pulled from manchester) only 281 of them are distinct.

[cegme]

Are these tweetids also duplicated in the index?

[austinpgraham]

No sir they are not.