The Spark PCA node had a configuration to choose the number of components however I would like to know the number of components to choose looking at the eigenvalues ( which are not given by the node). Is there a way I could do this?
Hi,
my apologies for the late response. The Spark PCA unfortunately does not provide eigenvalues as you noticed.
Spark's own PCA implementation only provides athe PCA transformation matrix (containing the eigenvectors if I understand correctly). This matrix is the second output porf of the Spark PCA node.
I am afraid I can only recommend you to use the Spark Java Snippet node to compute the eigenvalues from that.
Best,
Björn
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.