Tibco Spotfire File Writer

MaxSterling · March 11, 2019, 11:43am

Hi can someone help me in troubleshooting the Spotfire File Writer node.
I have a 7GB SBDF file that i’m trying to write to the Spotfire library, however it takes long more than 8 hours and it does not get out of the executing state on 50% never. I tried with a smaller example of 80 rows and it takes 1.5 minutes to upload, and the node completes successfully. So it seems to me that some transfer might be happening but really slow, however I don’t know how to prove it.

When i check the knime log I find the following

2019-03-11 09:01:18,282 : DEBUG : main : ExecuteAction :  :  : Creating execution job for 1 node(s)...
2019-03-11 09:01:18,283 : DEBUG : main : NodeContainer :  :  : TIBCO Spotfire File Writer 0:144 has new state: CONFIGURED_MARKEDFOREXEC
2019-03-11 09:01:18,283 : DEBUG : main : NodeContainer :  :  : TIBCO Spotfire File Writer 0:144 has new state: CONFIGURED_QUEUED
2019-03-11 09:01:18,284 : DEBUG : main : NodeContainer :  :  : wf_P&L_SMRY_MAIN 0 has new state: EXECUTING
2019-03-11 09:01:18,284 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer :  :  : ROOT  has new state: EXECUTING
2019-03-11 09:01:18,285 : DEBUG : KNIME-Worker-29 : WorkflowManager : TIBCO Spotfire File Writer : 0:144 : TIBCO Spotfire File Writer 0:144 doBeforePreExecution
2019-03-11 09:01:18,285 : DEBUG : KNIME-Worker-29 : NodeContainer : TIBCO Spotfire File Writer : 0:144 : TIBCO Spotfire File Writer 0:144 has new state: PREEXECUTE
2019-03-11 09:01:18,285 : DEBUG : KNIME-Worker-29 : WorkflowManager : TIBCO Spotfire File Writer : 0:144 : TIBCO Spotfire File Writer 0:144 doBeforeExecution
2019-03-11 09:01:18,285 : DEBUG : KNIME-Worker-29 : NodeContainer : TIBCO Spotfire File Writer : 0:144 : TIBCO Spotfire File Writer 0:144 has new state: EXECUTING
2019-03-11 09:01:18,285 : DEBUG : KNIME-Worker-29 : WorkflowDataRepository : TIBCO Spotfire File Writer : 0:144 : Adding handler cae8f824-af9c-4160-8652-cdb6177e36ec (TIBCO Spotfire File Writer 0:144: <no directory>) - 341 in total
2019-03-11 09:01:18,285 : DEBUG : KNIME-Worker-29 : LocalNodeExecutionJob : TIBCO Spotfire File Writer : 0:144 : TIBCO Spotfire File Writer 0:144 Start execute
2019-03-11 09:01:18,285 : DEBUG : KNIME-Worker-29 : SpotfireFileWriterNodeModel : TIBCO Spotfire File Writer : 0:144 : Entering execute(inData, exec) of class SpotfireNodeModel.
2019-03-11 09:01:18,286 : INFO  : KNIME-Worker-29 : SpotfireFileWriterNodeModel : TIBCO Spotfire File Writer : 0:144 : Filename: http://xx.xx.xx.xx:xx/spotfire/xxx/xxx/xxx/xxx/MOAB.sbdf
2019-03-11 09:01:18,286 : INFO  : KNIME-Worker-29 : SpotfireFileWriterNodeModel : TIBCO Spotfire File Writer : 0:144 : URL: http://xx.xx.xx.xx:xx/spotfire/xxx/xxx/xxx/xxx/MOAB.sbdf
2019-03-11 09:01:18,290 : INFO  : KNIME-Worker-29 : SpotfireFileWriterNodeModel : TIBCO Spotfire File Writer : 0:144 : Writing TIBCO Spotfire file: D:\AppData\Local\Temp\2\knime-spotfire3915972047743362732.sbdf
2019-03-11 09:01:18,354 : DEBUG : KNIME-Worker-29 : AbstractTableStoreReader : TIBCO Spotfire File Writer : 0:144 : Opening input stream on file "D:\AppData\Local\Temp\2\knime_wf_P_L_SMRY_MAI61614\knime_container_20190309_237446741875518294.bin.gz", 0 open streams
2019-03-11 09:01:18,377 : DEBUG : KNIME-Worker-29 : AbstractTableStoreReader : TIBCO Spotfire File Writer : 0:144 : Opening input stream on file "D:\AppData\Local\Temp\2\knime_wf_P_L_SMRY_MAI61614\knime_container_20190309_387987495252945984.bin.gz", 0 open streams
2019-03-11 09:01:18,389 : DEBUG : KNIME-Worker-29 : AbstractTableStoreReader : TIBCO Spotfire File Writer : 0:144 : Opening input stream on file "D:\AppData\Local\Temp\2\knime_wf_P_L_SMRY_MAI61614\knime_container_20190309_1648354879695091752.bin.gz", 0 open streams
2019-03-11 09:01:18,390 : DEBUG : KNIME-Worker-29 : AbstractTableStoreReader : TIBCO Spotfire File Writer : 0:144 : Opening input stream on file "D:\AppData\Local\Temp\2\knime_wf_P_L_SMRY_MAI61614\knime_container_20190309_912066590750018952.bin.gz", 0 open streams
2019-03-11 09:01:18,391 : DEBUG : KNIME-Worker-29 : AbstractTableStoreReader : TIBCO Spotfire File Writer : 0:144 : Opening input stream on file "D:\AppData\Local\Temp\2\knime_wf_P_L_SMRY_MAI61614\knime_container_20190309_3634024237826621962.bin.gz", 0 open streams
2019-03-11 09:01:18,391 : DEBUG : KNIME-Worker-29 : AbstractTableStoreReader : TIBCO Spotfire File Writer : 0:144 : Opening input stream on file "D:\AppData\Local\Temp\2\knime_wf_P_L_SMRY_MAI61614\knime_container_20190309_9117868408124061688.bin.gz", 0 open streams
2019-03-11 09:01:40,464 : DEBUG : KNIME-Worker-29 : AbstractTableStoreReader : TIBCO Spotfire File Writer : 0:144 : Closing input stream on "D:\AppData\Local\Temp\2\knime_wf_P_L_SMRY_MAI61614\knime_container_20190309_237446741875518294.bin.gz", 0 remaining
2019-03-11 09:01:50,288 : DEBUG : KNIME-Worker-29 : AbstractTableStoreReader : TIBCO Spotfire File Writer : 0:144 : Closing input stream on "D:\AppData\Local\Temp\2\knime_wf_P_L_SMRY_MAI61614\knime_container_20190309_387987495252945984.bin.gz", 0 remaining
2019-03-11 09:10:20,218 : DEBUG : KNIME-Worker-29 : AbstractTableStoreReader : TIBCO Spotfire File Writer : 0:144 : Closing input stream on "D:\AppData\Local\Temp\2\knime_wf_P_L_SMRY_MAI61614\knime_container_20190309_1648354879695091752.bin.gz", 0 remaining
2019-03-11 09:13:44,367 : DEBUG : KNIME-Worker-29 : AbstractTableStoreReader : TIBCO Spotfire File Writer : 0:144 : Closing input stream on "D:\AppData\Local\Temp\2\knime_wf_P_L_SMRY_MAI61614\knime_container_20190309_912066590750018952.bin.gz", 0 remaining
2019-03-11 09:13:59,896 : DEBUG : KNIME-Worker-29 : AbstractTableStoreReader : TIBCO Spotfire File Writer : 0:144 : Closing input stream on "D:\AppData\Local\Temp\2\knime_wf_P_L_SMRY_MAI61614\knime_container_20190309_3634024237826621962.bin.gz", 0 remaining
2019-03-11 09:14:28,652 : DEBUG : KNIME-Worker-29 : AbstractTableStoreReader : TIBCO Spotfire File Writer : 0:144 : Closing input stream on "D:\AppData\Local\Temp\2\knime_wf_P_L_SMRY_MAI61614\knime_container_20190309_9117868408124061688.bin.gz", 0 remaining
2019-03-11 09:14:28,761 : INFO  : KNIME-Worker-29 : SpotfireFileWriterNodeModel : TIBCO Spotfire File Writer : 0:144 : Uploading to library with the file: /spotfire/xxx/xxx/xxx/xxx/MOAB.sbdf
2019-03-11 09:14:28,761 : INFO  : KNIME-Worker-29 : SpotfireFileWriterNodeModel : TIBCO Spotfire File Writer : 0:144 : Uploading TIBCO Spotfire file: D:\AppData\Local\Temp\2\knime-spotfire3915972047743362732.sbdf
2019-03-11 09:14:28,762 : INFO  : KNIME-Worker-29 : SPOWebServiceClient : TIBCO Spotfire File Writer : 0:144 : SPOWebServiceClient url: http://xx.xx.xx.xx:xx username: xxxxxxx password: ****
2019-03-11 09:14:28,763 : INFO  : KNIME-Worker-29 : SPOWebServiceClient : TIBCO Spotfire File Writer : 0:144 : Using URL: http://xx.xx.xx.xx:xx
2019-03-11 09:15:13,222 : INFO  : KNIME-Worker-29 : SPOServletServiceClient : TIBCO Spotfire File Writer : 0:144 : HTTP Authentication Basic for SPOServletServiceClient xxxxxxxx ****
2019-03-11 09:15:13,224 : INFO  : KNIME-Worker-29 : SPOServletServiceClient : TIBCO Spotfire File Writer : 0:144 : Servlet client URL: http://xx.xx.xx.xx:xx/spotfire/attachment
2019-03-11 09:15:57,374 : INFO  : KNIME-Worker-29 : SpotfireFileWriterNodeModel : TIBCO Spotfire File Writer : 0:144 : Falling back to Out-of-Process file uploader
2019-03-11 09:15:57,376 : INFO  : KNIME-Worker-29 : SpotfireFileWriterNodeModel : TIBCO Spotfire File Writer : 0:144 : D:\KNIME\plugins\com.tibco.spotfire.libraryclient.oop_1.8.20
2019-03-11 09:15:57,376 : INFO  : KNIME-Worker-29 : SpotfireFileWriterNodeModel : TIBCO Spotfire File Writer : 0:144 : "D:\KNIME\plugins\com.tibco.spotfire.libraryclient.oop_1.8.20\bin;D:\KNIME\plugins\com.tibco.spotfire.libraryclient.oop_1.8.20\lib\*"
2019-03-11 09:15:57,376 : INFO  : KNIME-Worker-29 : SpotfireFileWriterNodeModel : TIBCO Spotfire File Writer : 0:144 : command line: "D:\KNIME\plugins\org.knime.binary.jre.win32.x86_64_1.8.0.152-01\jre\bin\java.exe" -classpath "D:\KNIME\plugins\com.tibco.spotfire.libraryclient.oop_1.8.20\bin;D:\KNIME\plugins\com.tibco.spotfire.libraryclient.oop_1.8.20\lib\*" com.tibco.spotfire.libraryclient.FileUploader http://xx.xx.xx.xx:xx/spotfire/xxx/xxx/xxx/xxx/MOAB.sbdf xxxx xxxxxx== "D:\AppData\Local\Temp\2\knime-spotfire3915972047743362732.sbdf"
2019-03-11 09:15:57,376 : INFO  : KNIME-Worker-29 : SpotfireFileWriterNodeModel : TIBCO Spotfire File Writer : 0:144 : [D:\KNIME\plugins\org.knime.binary.jre.win32.x86_64_1.8.0.152-01\jre\bin\java.exe, -classpath, "D:\KNIME\plugins\com.tibco.spotfire.libraryclient.oop_1.8.20\bin;D:\KNIME\plugins\com.tibco.spotfire.libraryclient.oop_1.8.20\lib\*", com.tibco.spotfire.libraryclient.FileUploader, http://xx.xx.xx.xx:xx/spotfire/xxx/xxx/xxx/xxx/MOAB.sbdf, xxxx, xxxxxx==, "D:\AppData\Local\Temp\2\knime-spotfire3915972047743362732.sbdf"]

wiswedel · April 4, 2019, 6:47am

(That’s a node contributed by Tibco who is a KNIME partner.)

According to the log file the node execution triggers another process (last line in the log). Can you identify that process in the system’s process table? If so, can you extract a jstack so that we (or Tibco) can figure out what’s going on? (I am lazy here and refer to another thread that nicely describes how to use ‘jstack’).

I’d also like to copy @andrewberridge here. Andrew is with Tibco and can possibly help?!

Thanks,
Bernd

andrewberridge · April 12, 2019, 9:46am

Hi there and sorry for the delay in answering this post. I think that 7Gb is probably too big for the Spotfire library to accept and so I would suggest you find another way of using the file. You could always save it to a shared folder on a filing system and enable access to that file. If you need to load the file on the web player, you’ll need to configure the web player to allow access to the file path that contains it.
I would suggest using Scheduled Updates to load the analysis into the memory of the web player so that it’s immediately available to end users.
Bernd is right - the node is executing the file upload out of process - this isn’t ideal as you won’t get any progress indication beyond 50% until the file upload is complete. The out of process uploader is a fallback solution used when other methods fail. I will be releasing a new version of the nodes in the next few months, so please follow the partner forum to be kept up to date with this. You can always look me up on LinkedIn too and connect with me there and send me a message in order to be in direct contact with me.
Andrew

MaxSterling · April 12, 2019, 12:48pm

Thanks Andrew, I suspected it would be too heavy for library, do you have a reference how can i set up the web player to resolve the shared location? I tried to make a POC by having a folder shared to “Everyone” but I get a message

“Some part of the data table could not be loaded. A data source may be missing or has been changed.”

Best

andrewberridge · May 2, 2019, 11:06am

Hi Max,

I’ve been out for a while, sorry again for the delay. The Web Player documentation gives the information here:

https://support.tibco.com/s/article/How-to-configure-access-to-local-and-shared-network-directories-for-the-TIBCO-Spotfire-Web-Player

Andrew