Unintended 4095 Character/Byte limit on REST Web Services/POST Request Node?

I'm having issue sending any body larger than 4095 characters/bytes using POST Request Node (KNIME Labs > REST Web Services > POST Request).  My server program is showing no data when body larger than 4095 bytes are sent.

I was able to send much larger raw data using Postman, and somthing similar is mentioned but nothing about how big the data can be right here: https://tech.knime.org/forum/knime-general/post-request-body-binary

Does anyone have the same/similar issue?  Is this a bug?

I'm using KNime 3.3.2 full install.

 

 

 

 

 

I haven't heard of this behavior before. However it can happen that the node sends chunked requests if the data gets larger and your webservice isn't able to deal with them.

I tested the webservice using both Java program (handrolled) and Postman, it can handle huge chunks of data just fine.  It's likely the node.

I created a Java snippet that can send the data as well, and that worked (I'll post it if that helps).  I'll probably have to create a custom node for this, but hoping someone has a better workaround.

The REST nodes are perfectly capable of sending more than 4kB of data (I just verified it, just to be 100% sure). Therefore it must be an issue with your configuration or web service. You could sniff the request that is sent by the node and check its contents.

Thanks for checking!  At least I know that POST node works in another machine.

The webservice test code is dead simple and it can receive over 4kB data for sure (posted below), and I was able to send large data using KNIME's java snippet (also posted below).

 

If you can send > 4kB data in your POST node, maybe:

1. I could be misconfiguring the node, or

2. using a different version of KNIME, or

3. There may a difference in which POST request is being sent by POST request node and the Java snippet node.

I'm thinking it's 3.  Like you said, I'll check what server is receiving.

 

 

=====================================================

Here's the webservice test code in python Flask:

from flask import Flask

@app.route('/testinput', methods=['POST'])
def testinput:
	binary_data = request.get_data()
	return "inputlen_"+str(len(binary_data))+"_end"

if __name__ == "__main__":
  app.run(debug=True, host='0.0.0.0', port=8000)

 

=====================================================

Here's KNIME Java snippet code.

  1. Input
    1. flowvar "url" is the URL of the web service
    2. input table is single row with a BLOB column named "body"
  2. Output
    1. Single row table with column "abc" which has the output of the webservice
      1. from this snippet: 
      2. abc_10864_def

 

// system imports
import org.knime.base.node.jsnippet.expression.AbstractJSnippet;
import org.knime.base.node.jsnippet.expression.Abort;
import org.knime.base.node.jsnippet.expression.Cell;
import org.knime.base.node.jsnippet.expression.ColumnException;
import org.knime.base.node.jsnippet.expression.TypeException;
import static org.knime.base.node.jsnippet.expression.Type.*;
import java.util.Date;
import java.util.Calendar;
import org.w3c.dom.Document;

// imports for input and output fields
import java.io.InputStream;

// Your custom imports:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;

import java.util.List;
import java.util.ArrayList;
import java.util.stream.Collectors;

//import java.io.IOUtils;
// system variables
public class JSnippet extends AbstractJSnippet {
  // Fields for input columns
/** Input column: "body" */
  public InputStream c_body;
  // Fields for input flow variables
/** Input flow variable: "url" */
  public String v_url;

  // Fields for output columns
/** Output column: "abc" */
  public String out_abc;

// Your custom variables:
	
// expression start
    public void snippet() throws TypeException, ColumnException, Abort {
// Enter your code here:

		try {
			URL url = new URL(v_url);
			HttpURLConnection conn = (HttpURLConnection)url.openConnection();

			List<Byte> cBodArry = new ArrayList<Byte>();
			
			Reader r = new BufferedReader(new InputStreamReader(c_body));
			for(int c; (c = r.read()) >= 0;)
				cBodArry.add((byte)c);

			
			byte[] postDataBytes = new byte[cBodArry.size()];

			int i=0;
			for(byte b: cBodArry)
				postDataBytes[i++] = b;
			InputStreamReader reader = new InputStreamReader(c_body);


			conn.setRequestMethod("POST");
			conn.setRequestProperty("content-type", "application/x-www-form-urlencoded");
			conn.setRequestProperty("Content-Length", String.valueOf(postDataBytes.length));
			conn.setDoOutput(true);
			conn.getOutputStream().write(postDataBytes);
			
			BufferedReader in = new BufferedReader(new InputStreamReader(conn.getInputStream(), "UTF-8"));
			
			out_abc = in.lines().collect(Collectors.joining(""));
		} catch (MalformedURLException e) {
			e.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		}



// expression end
    }
}

Sorry for reviving a relatively old thread. I am experiencing exactly the same issue. I am sending a POST request through the POST Request KNIME node. When the body of the request is 4095 bytes or less, all good. As soon as the body size is >4095 bytes, the POST request has an empty body.

It would be great if you could look into this as it seems to be an issue connected with the POST Request KNIME node. If I send the samne data through a Python script using the "request" module, everything works perfectly as expected, no matter what size the data chunk is.

Thank you very much, best regards

Paolo

I am using KNIME 3.4.0.

The problem can be easily reproduced setting up a simple Python REST server:

#!/usr/bin/env python

import flask
from flask import Flask, jsonify

app = Flask(__name__)

from flask import request

@app.route('/rest', methods=['PUT'])
def test_put():
    return jsonify(request.json), 201

@app.route('/rest', methods=['POST'])
def test_post():
    return jsonify(request.json), 201

if __name__ == '__main__':
    app.run()

And then sending a POST Request to that REST server from KNIME. If the body is larger than 4095 bytes, a 400 is returned, otherwise you get a 201, as expected.

This isn't a problem with the node but with flask in Python which doesn't support chunked requests. See http://cxf.apache.org/docs/client-http-transport-including-ssl-support.html#ClientHTTPTransport(includingSSLsupport)-ANoteAboutChunking for an explanation of why chunked requests are used.

Thanks for your quick reply, that was helpful. Is there a way to switch off chunking, by any chance?

Currently there is no way to disable chunking. I can open an enhancement request but I cannot give you an estimate when it will be implemented.

Thank you!

Can I second this enhancement request, please? Currently Apache does not support chunked encoding to CGI scripts, so any web service delivered via an Apache FCGI backend will not work with the POST node.

Dear Thor,

Sorry for reviving again an old thread.
it would be really useful to us if you could add a checkbox to disable chunking on the POST and PUT nodes.

Currently we need to use the Palladian HttpRetriever nodes to do POST and PUT, as it does not do chunking.
The drawback of the HttpRetriever node is that it does not support workflow credentials, so the only way to do basic authentication is to add a Authentication header with base64-encoded user:password, which works fine.
However, we need to use a Java Snippet to do the base64-encoding of user credentials, and as the Java Snippet node cannot access workflow credentials, the credentials need to be entered in clear text as workflow variables, as described here:

https://www.knime.com/book/http-nodes

As you know, storing credentials in clear text is unpleasant.

In conclusion, in would be very highly appreciated if you could either

  • add an option to disable chunking on PUT/POST nodes

or, alternatively

  • add a tab to the HttpRetriever node to use workflow credentials for authentication, as for the GET/PUT/POST nodes

Thank you very much in advance for your time and consideration,
kind regards

Paolo