Hello,
I’m trying to post a datafile in csv format utilizing an API. I can get it to work in CURL but I’m struggling to translate it to KNIME. The API I have been trying to post to is CDD or Collaborative Drug Discovery. I’ve seen a few threads and workflows that attempt this but It’s not clear if any of these prior cases have been successful.
The scenario is that there is a mapping template that had project info, maps the column headers for uploading into the database etc.
I thought OK, maybe something is wrong with my file so I stripped it all back to a very simple file with 4 columns etc. but I am getting the same results.
I’m really stuck and I’m not savvy enough to know how to find the illegal parameters.
Any help would be super appreciated and I plan to put the solution on the HUB so others can benefit. I’ve uploaded a modified version of a workflow that @qqilihq has previously developed. Palladium_HTTP_Retriever_Multipart_Example_JMO.knwf (35.3 KB)
Just dropped you a line at the email you provided. I looked at the creator node and I patterned it after you image example. You took the URI of the image and went from file to binary object. I did the same with my csv file.
I have just replied via email, but I thought I keep this thread up-to-date as well. At the end, it was indeed the parameter name which needed to be changed:
Thanks @qqilihq Very interesting and tricky… As that “Name” auto populates when the input column is selected. I wonder how many different names there could be that would need to be set when they are wrapped / converted into a BinaryObject.
Anyway, look forward to posting this version of your workflow on the HUB so others can benefit!
The only trick left is how to properly pass the file name information to the HTTP Entity Creator node from a variable. I have tried list and set and string (set shown here) → [file_0.csv, none.json]
but if throws the following error.
{ “error” : “no implicit conversion of ActionDispatch::Http::UploadedFile into String”, “code” : 400 }
Passing the file name is very useful as that is recorded by the application and allows for checking to make sure the right files get uploaded.
@qqilihq Thanks for looking into this again! I did try both of those but not with the single quotes. I think I tried double quotes. I’ll try now and reply back.
Filename value comes in, java snippet adds the single quotes around it, constant value adds the ‘null’, null, or ‘’ and then the list is built using the column aggregator and then passed as a variable to the muliti creator node and the variable is selected for filename.
I have just inspected this in our example workflow which we shared before. To figure out the proper structure of the flow variables, I first used the configured node and passed the data into a newly created flow variable (if anybody who reads this actually wonders what these fields without any title are for: these allow to create a new flow variable based on data entered in the dialog ).
So, I created a variable fileNames to output the current configuration:
So the node should allow to pass a string array (if I interpret the icon correctly) flow variable, where empty fields from the dialog are just filled with null.
Thus, I assume that the structure or data type which you feed to the node through the variable is probably wrong. Could you @j_ochoada show a screenshot of your “Table Row to Variable” output with the “Flow Variables” tab? And also your “Flow Variables” configuration of the Multipart Encoded HTTP Entity Creator node?
I just had the opportunity to try this locally, and the problem is, that the “Create String Array Variable” meta node will obviously not allow null values. If you enter “null”, it will be a literal “null” (as a string, and not as null).
The Table Row to Variable node which I tried next insists on replacing missing values with a string placeholder.
Other nodes (such as the Java snippet ones) do not even seem to support array-typed flow variables. Not really satisfying and KNIME probably needs to do some more of their homework to finish that half-baked flow variable support
So, the question: How can we create a string array flow variable with null values? I have no clue to be honest, so we’re somewhat back to zero now.
Fortunately I came up with a different solution:
Leave the Multipart Encoded HTTP Entity Creator node as it is. No flow variables. Just configure a static placeholder filename such as file.svg as this point.
After this node, I put a Binary Objects to Strings node to temporarily convert the BLOB into a string. For decoding I set ISO-8859-1 (other ones will destroy the payload!).
This enables you to manipulate the payload using a String Manipulation node. At this point, you can put an expression like so which will set the desired filename: replace($multipartEntity$, "filename=\"file.svg\"", "filename=\"renamed.svg\"") (backslashes are needed to escape the inner " quotes).
Afterwards you can transform the string back to a BLOB using a Strings to Binary Objects node. Ensure to use ISO-8859-1 as well.
From there on, continue as before and input the result into the HTTP Retriever.
Hey @qqilihq ! Wow thanks so much for continuing to keep on this. Bummer on one hand but fantastic that there is a work around. I’m glad I wasn’t making a careless mistake. But on the other hand, that would have been much easier.
).
