@Saishiyam how large is the data of you just write it to a KNIME table. How large is the Excel file if you say append.
Have you tried 6 GB? Leave some for the system.
Try a workflow that just does the export and utilize Garbage collection. Cache the data before export.
You could try a newer KNIME version although there seem to be problems with the very latest version of the Excel writer. So maybe look for version 4.5.3 and a solution to this ticket.
Next thing could be to try R or python openpyxl.