@takbb I have a lot of fields that are getting ignored by the KNIME Table Session Connector and even though I have renamed them and cleaned them up (removed ones with missing values) they are not appearing on the available fields.
One that is important is a column with email addresses.
The source of the data is a KNIME table file and many of the fields come over fine, but this email string field is getting ignored. I even just tried to bring over a few fields and this one keeps getting missed. Could it be due to a value in one of the rows? Honestly, not sure how to troubleshoot why this and other fields are getting missed.
The field name was renamed to EMAIL to make sure that wasn’t the issue.
Hi @aehrenworth , what’s the data type of the columns you are having problem with? The KNIME Table Session Connector component limits the columns that you can select to a subset of datatypes, as they must be compatible with the underlying sql database. e.g. it should allow String, Integer, Long, Double, Local DateTime, Local Date and Boolean. It’s possible that the component is being over-zealous in disallowing columns of type I hadn’t considered that would still work with the database. Columns of other data types to the above list, or with no datatype will not be visible in the config.
Can you upload an example workflow, with just a Table Creator attached to a KNIME Table Session Connector.
Manually place some sample data of the form that you are having problems with, and have it passed to the KNIME Table Session Connector, ensuring that it exhibits the problem you are having, and I can take a look.
Can you also upload screenshots of the input data table being fed to the component, the component config, plus component output data.
thanks for trying the component!
edit: One thing to note is that column names containing spaces will be converted to underscores, and some potentially incompatible characters in column names can be removed/modified by the component, but I think you are saying that the column doesn’t get created at all, rather than it doesn’t appear with the same name,
e.g. in this example the spaces in the names are converted to underscores…
In my example, my column that is getting missed already has an underscore “WORK_EMAIL”.
Oddly, when I tried to create sample data (replacing real emails with fake email names) and then passed this file into the node, it worked fine.
Not sure how to share the actual list (something in the data or in that field config) since it has real email addresses.
I did a little more testing just now, and if I sampled for 10 records randomly, sometimes it was working and sometimes it was missing the field. That makes me think some of the fields might have special characters that are causing the component to fail. I did String cleaner but that didn’t fix it.
Very perplexed and not sure how to give you what you need so you can apples to apples compare.
Even weirder is if I sample for the first 50 records then I keep upping it past 60 then this behavior comes up. but when I look at the data in the field itself nothing stands out that would cause this to happen.
I then tried filtering one row and if I sample less than 50 the field appears in the node… but if I sample 75 or 100 and still filter for one row the node misses the field.
Not sure what is going on but can’t really share the exact data I am working with.
Another piece of info: I tried using a CSV version of the data versus a KNIME table file and produced the same behavior. going to see if there is any discerning metadata for the fields being ignored.
Last thing I just noticed — this setting is likely what is causing the fields to be ignored. but not quite sure what would cause a field to not have a domain.
using the Domain Calculator for one of the fields made it appear in the Table Session node.
I’ve done further investigation and based on your earlier comment, I’ve replicated the problem. I was originally going to consider adding a config setting to override the “columns without domain” setting, but have determined that actually there is really no reason I can think of for which leaving this switched on would be beneficial, so I have now rebuilt this component (and it’s sister component KNIME Table Connector),with that switched off internally.
Please update or re-download your component, and let me know if it is now ok.
thanks again for trying the component, and reporting this issue.
yes, thank you so much. Appreciate the dialog here.
I much prefer to use the SQL query option for filtering this data on the variety of use cases I might have and I want to run the larger DB pull once each day and then query off the static info (if that makes sense).
This is a great option for doing that so really appreciate you efforts to create it!
Yes, I confirm your adjustment is allowing this to work without the Domain Calculator.
