Slack nodes for KNIME

raghum · September 14, 2020, 2:34pm

Title (username) is not working at for Channels. However, it works for User.

swebb · September 14, 2020, 10:22pm

I have just checked setting the title with the ‘Message Slack channel’ and the 'Message Slack channel (row based) in KNIME 4.1.3 and 4.2.0 and the most recent slack node version (2.6.1). It all seems to be working fine my end.

The node documentation states:

Notes on username: By default the message will be sent and display with the apps name. If the chat:write:customize scope has been added then it is possible to set a username parameter. If changing the display name / username of the source of the message is desirable then check ‘Set username’ and provide the username to post the message as.

Can you check you have the scope chat:write:customize added to your app?

You didn’t provide any further information on your first issue (no message received) so I’m not really sure where to go from there. After double checking the documentation it looks like we would expect to see an error in the response if it is caused by Rate Limiting. It’s possible the Message Slack channel hides this as it doesn’t return the response from Slack in the node output. It’s possible the node is swallowing this so I’ve added some additional debug output which will be viewable in the console or log output if you set the level to DEBUG. An example:

DEBUG Message Slack channel (row based) 0:3 Slack node class: class com.sjwebb.knime.slack.api.SlackBotApi
DEBUG Message Slack channel (row based) 0:3 Response: ChatPostMessageResponse(ok=true, warning=null, error=null, needed=null, provided=null, deprecatedArgument=null, responseMetadata=null, channel=G011DQKBL9H, ts=1600120638.002600, message=Message(type=message, subtype=null, team=T011N55HK8C, channel=null, user=U011HRU0DSA, username=null, text=demo-private, blocks=null, attachments=null, ts=1600120638.002600, threadTs=null, intro=false, starred=false, wibblr=false, pinnedTo=null, reactions=null, botId=B011HRU085C, botLink=null, displayAsBot=false, botProfile=BotProfile(id=B011HRU085C, deleted=false, name=KNIME Message Bot, updated=1586471000, appId=A011HK9LK9Q, icons=BotProfile.Icons(image36=https://a.slack-edge.com/80588/img/plugins/app/bot_36.png, image48=https://a.slack-edge.com/80588/img/plugins/app/bot_48.png, image72=https://a.slack-edge.com/80588/img/plugins/app/service_72.png), teamId=T011N55HK8C), icons=null, file=null, files=null, upload=false, parentUserId=null, inviter=null, clientMsgId=null, comment=null, topic=null, purpose=null, edited=null, unfurlLinks=false, unfurlMedia=false, threadBroadcast=false, replies=null, replyCount=null, replyUsers=null, replyUsersCount=null, latestReply=null, subscribed=false, xFiles=null, lastRead=null, root=null, itemType=null, item=null))
DEBUG Message Slack channel (row based) 0:3 Chat message isOk: true
DEBUG Message Slack channel (row based) 0:3 Chat message errors: null
DEBUG Message Slack channel (row based) 0:3 Chat message warning: null

This update is in the KNIME nightly update site and the version number is 2.6.2.

swebb · September 14, 2020, 10:57pm

I have now also added rate limiting to the nodes that send messages (User and Channel). This will be available in version 2.7.0 when this has built and made it into the nightly/trunk update site.

Cheers

Sam

raghum · September 15, 2020, 4:15pm

ok, will check back later today.

raghum · September 16, 2020, 5:17pm

Sam, I tested and the title is working for Slack channel.

Now, I am getting following errors very frequently using Get Users node

ERROR Get Users 3:0:0:43:5 Execute failed: status: 429, error: ratelimited, needed: null, provided: null, warning: null
ERROR Message Slack User (beta) 3:0:0:43:6 SlackApiException(response=Response{protocol=http/1.1, code=429, message=Too Many Requests, url=https://slack.com/api/users.list}, responseBody={“ok”:false,“error”:“ratelimited”}, error=SlackApiErrorResponse(ok=false, warning=null, error=ratelimited, needed=null, provided=null))

ERROR Message Slack User (beta) 3:0:0:43:6 Execute failed: status: 429, error: ratelimited, needed: null, provided: null, warning: null

Any idea on this error?

Is it possible to overcome ratelimited error at your end by following this way (https://medium.com/slack-developer-blog/handling-rate-limits-with-slacks-apis-f6f8a63bdbdc)?

And I dont understand why are we getting ratelimited error on Get Users node.

swebb · September 16, 2020, 5:44pm

Here’s some more info on rate limits: Rate Limits | Slack

The Java SDK I use has some functionality for managing rate limits which is what I started using for the 2.7.0 update. Is this the version you were seeing those errors? Looks like I need to do a bit work on managing this. I fear the KNIME nodes may have been swallowing some of these errors before and that may be why you weren’t seeing your messages come through.

For reference what kind of behaviour to you have with KNIME sending slack messages? The occasional message, bursts of a few messages, lots of constant messaging?

And I dont understand why are we getting ratelimited error on Get Users node.

I believe rate limits apply both to the method being called as well at the bot. So if you’re bot has been rate limited due to too many calls to a chatPostmessage method then all methods.

The get users node calls users.list method | Slack which has a rate limit of 20 calls per minute. It looks like I may have been wrong about a global app rate limit

raghum · September 16, 2020, 6:09pm

For now, we are just testing with one user and getting these errors very often. In realtime scenario, we will not be bursting many messages at one shot but we might run these nodes in parallel. Probably, ~10 workflows in parallel.

swebb · September 16, 2020, 6:11pm

I’m applying some updates to the rate limit handling now which will hopefully catch some of these issues. Will let you know when it’s available.

swebb · September 16, 2020, 11:08pm

I have added some additional behaviour:

Ensure that all nodes using the same token use the same SlackBotApi instance thus ensuring they use the same MetricStore to back the Rate Limit monitoring
Added Rate Limit respecting behaviour into the Get Users node
Added a new node Slack API Metrics which can get the current state of the Metric Store backing the SlackApiBot instances
Fixed a potential bug where the Message Slack Channel (row based) may have been using the token in the Preferences page rather than the token in the node dialog if overridden.

Example output of the new node:

I set off a test of 7 nodes in parallel trying to post 100 messages each. Out of 181 messages I only got 1 warning:

These updates are available in version 2.8.0 in the nightly/trunk build as soon as I can upload the fixed test workflow (something seems to have gone wrong with my connection to the KNIME test server).

I’ve got the update site building, should be available shortly. Let me know if you see an improvement in behaviour or no change?

Cheers

Sam

raghum · September 28, 2020, 6:42pm

Sam,

I am still getting following error on top of latest version.

ERROR Get Users 2:15:0:43:5 Execute failed: status: 429, error: ratelimited, needed: null, provided: null, warning: null

Can we restrict to get for specified user(s) rather than getting whole list? I am just thinking to overcome this issue by this way.

swebb · September 28, 2020, 7:39pm

Ah, do you get this from a single call of the node?

I will provide you some more settings and maybe we can tune it to give you the information you need without rate limiting. It gets 200 users at a time at the moment so maybe if you have a lot of users it’s making too many requests. I thought it would manage it by itself but it doesn’t look like it.

It does look like there is an API call to get the details for a specific user so I will also look into a mode for that.

swebb · September 28, 2020, 11:06pm

I have pushed the following changes (still building):

Created a Slack Users Lookup By Email node

This new node take a column of email addresses and makes an API call to https://api.slack.com/methods/users.lookupByEmail. This will return the same information as the Get Users node but only for the users you specify by their email. This node requires the `users:read.email’ scope.

Available from: 2.9.0 (trunk/nightly)

Message Slack User:

Only convert display names e.g. @Sam to unique identifiers such as U011XEN1RDW once per execution. Before this was done for each row that was sent

This should reduce the number of API calls made. However, for the largest reduction you should provide the unique identifier directly so the node doesn’t have to look this up (tricky if the Get Users node doesn’t run though, but you can look it up from the slack application).

Available from: 2.9.1 (trunk/nightly)

I will look at adding an optional delay to the Get Users node between consecutive calls in the future but for today these are the changes and they may help.

Cheers

Sam

raghum · October 1, 2020, 9:08pm

I still see 2.8 version but not 2.91. version.

swebb · October 1, 2020, 9:51pm

Ah my mistake I only bumped the plugin version number but it’s the feature one that shows on the update site.

I think this version contains all the updates:

I have however just bumped the feature version number also (2.9.2) and it will rebuild shortly, might be better to wait until you see the version number update just to be sure. KNIME will need to be restarted before checking if it is currently open as it caches the update site contents.

Cheers

Sam

swebb · October 1, 2020, 10:24pm

I should now be available. Let me know how you get on.

Cheers

Sam

raghum · October 7, 2020, 7:15am

Sam, I have installed latest version but still dont see the new node (Created a Slack Users Lookup By Email node) as you mentioned.

Am I missing something?

Screen Shot 2020-10-07 at 12.12.48 AM

swebb · October 7, 2020, 9:10am

I just installed the latest version from

KNIME Community Extensions (Trusted) - https://update.knime.com/community-contributions/trunk

It appears for me:

Which update site did you install from and what version number does he feature show?

Cheers

Sam

raghum · October 9, 2020, 6:16am

Sam, Look like Slack Users Lookup by Email node solves our problem and it is efficient compare to Get Users.

Thanks for all.

swebb · October 9, 2020, 7:40am

OK cool. I will migrate the new changes over onto the production builds.

Cheers

Sam

mnguyen · March 4, 2021, 11:55pm

Hi @swebb, I love the Slack Integration nodes, I’m testing it on KNIME AP 4.1.3 with Slack Integration version 2.6.1.v202008132209 and it works great, but when I deploy it on KNIME server (4.10) with small license I’m getting “Execute failed: Provided channel name of xxxx-xxxx is not valid”. I also test using chat.postMessage method | Slack on the server and it works too. I’m not sure where else I can check below is the log, hoping you can help. Thanks.

04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput java.nio.file.FileSystemException: \chewy.local\common\public\knime\data_quality\slack_channels.csv: The process cannot access the file because it is being used by another process.
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:230)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at java.nio.file.Files.newOutputStream(Files.java:216)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.base.node.io.csvwriter.CSVWriterNodeModel.doIt(CSVWriterNodeModel.java:273)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.base.node.io.csvwriter.CSVWriterNodeModel.execute(CSVWriterNodeModel.java:243)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.NodeModel.execute(NodeModel.java:737)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.NodeModel.executeModel(NodeModel.java:571)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1236)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.Node.execute(Node.java:1016)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:557)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:218)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:124)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:334)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:210)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at java.util.concurrent.FutureTask.run(FutureTask.java:266)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
04-Mar-2021 18:33:42.485 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput ERROR KNIME-Worker-1235-Message Slack channel 192:7 Node Execute failed: Provided channel name of dq-private123 is not valid
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput com.sjwebb.knime.slack.exception.KnimeSlackException: Provided channel name of dq-private123 is not valid
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at com.sjwebb.knime.slack.nodes.messages.send.SendMessageNodeModel.execute(SendMessageNodeModel.java:49)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.NodeModel.execute(NodeModel.java:737)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.NodeModel.executeModel(NodeModel.java:571)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1236)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.Node.execute(Node.java:1016)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:557)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:218)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:124)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:334)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:210)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at java.util.concurrent.FutureTask.run(FutureTask.java:266)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
04-Mar-2021 18:33:42.563 WARNING [KNIME-RMI-syserr-50100] com.knime.enterprise.server.executor.rmi.ProcessOutputConsumer.logOutput at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
04-Mar-2021 18:33:42.578 INFO [KNIME Job Event Distributor [1]] com.knime.enterprise.server.executor.rmi.RMIJobStatusUpdaterImpl.updateInternalState Job: Slack Integration 2021-03-04 18.33.42 (565c619a-5602-47dd-b223-14e6fffb248d) of owner knimeadmin finished with state CONFIGURED
04-Mar-2021 18:33:43.766 SEVERE [http-nio-8080-exec-26] org.apache.catalina.core.StandardHostValve.invoke Exception Processing /knime/