WebSearcher node - new Twitter API?

Community Extensions Palladian & Selenium
#1

Hi Philipp,

It looks like Twitter have updated their API, as I'm getting an error when using previously OK workflows:

http://search.twitter.com/search.json?q=<theSearchString>&page=1&rpp=100&lang=en&result_typemixed

{"errors": [{"message": "The Twitter REST API v1 is no longer active. Please migrate to API v1.1. https://dev.twitter.com/docs/api/1.1/overview.", "code": 68}]}

 


(the other) Simon

#2

Hi Simon,

I also just discovered this yesterday :) I'm trying to have an update ready during the next days.

Philipp

#3

Hi Simon,

the next build of the update site (stable+nightly) contains a fixed TwitterSearcher supporting the required OAuth authentication scheme. Unfortunately, some more steps are now necessary for authenticating the API:

  1. Sign up for a Twitter account.
  2. Log in to Twitter Developers
  3. Go to My Applications
  4. Create a new application
  5. On the application page, click the 'Create my access token' button, wait some seconds and refresh the page
  6. Enter the following data in the KNIME settings: Consumer key, Consumer secret, Access token, Access token secret (in KNIME under Settings > KNIME > Palladian Web Searcher).

Best,
   Philipp

#4

Thanks Philipp,

Works again.

(the other) Simon

#5

Hi everyone,

i'm new to twitter-mining/paladian nodes in knime.
i got my account und put the settings right, but even example-querys from twitter api documentation just create an empty data table when WebSearcher is executed.

Do you have any idea or help/giude for me?

 

thanks in advance!

#6

Hi bar,

can you tell me, which queries you perform? Or even better, upload your example workflow, so that I can have a look?

Best,
Philipp

#7

Hi Philipp,

one of my test-queries is https://api.twitter.com/1.1/search/tweets.json?q=%23Bundesliga&result_type=recent.

my flow is pretty simple: TableCreator containing the query-string in col1, connected to a WebSearcher-Node.

 

 

 

#8

Hi bar,

tried, it works for me (sample workflow I used is attached). Are you using the most current version of the Palladian plugins? Do you get any log output during search in the ?

Best,
Philipp

#9

Problem solved! I was up to write the whole query-url by myself and not only putting the hashtag in column1...

Thank you very much!
Benjamin

#10

I've successfully used the Palladian WebSearcher node to query Twitter. I'd like to restrict the tweets and so entered the string "#fitbit&until=2013-12-08" in the column read by WebSearcher. But I get an empty table. 

Any suggestions how I can specify the parameters from the Twitter API found here: 

https://dev.twitter.com/docs/api/1.1/get/search/tweets

Thanks!

David

#11

Hi David,

I'm sorry, this is currently not supported by the nodes. As we have this functionality in the underlying Palladian library, we might eventually provide it with the nodes (including other search criteria), but currently, I can give no estimate when this will be the case.

Best,
Philipp

#12

Hi, I am using the Palladian Websearcher and also TextClassifier nodes in the class I'm teaxching. I wonder if you can help answer a couple of questions:

  1. On Websearcher node, how can I get the API keys etc for the other search engines e.g., Google Custom Search? I can get the Google Cusotm Search API key but what is the Google Custom Search Identifier?
  2. On the TextClassifier node, do you have the corpus for determining the polarity of terms and can one modify the list based on domain requirements? Any references?

Thanks.

Nathaniel Lin

#13

Hi Nathaniel,

  1. The "identifier" is the name of the search engine which you must create in advance in the Google Custom Search settings.
  2. Can you be a bit more specific here?

Best,
Philipp

#14

So if I want to do a Google custom search and have registered with Google. What would the two fields be, i.e.,  uner \KINME\Preference\KNIME\Palladian Web Searcher\ entries for Google Customer Seach API key and Google Custom Search identifier? Also under the node WebSearcher I will then use Google Custom Search for the Search engine drop-down slections, right?

#15

Excuse me, what I posted above was wrong. The identifier is of course not the "name" of your Google custom search engine, but the search engine ID. In my case, it's an alphanumeric string of 33 characters, starting with numbers, colon, then letters+digits. API key can be obtained from the Google Cloud Console.

Does that help?