question

There is a limit for the number of columns for a Statistics Collector, because that object stores its data in a Bundle node. The limit is actually based on the number of bytes per row. The maximum number of bytes per row is 65535. Each double column requires 8 bytes. So if all your columns are double types (which is the case for you), you can only have 8191 fields.

I'll add an issue to the dev list, to find a way to throw an exception if you hit the limit.

By contrast, the limit on the number of rows in a bundle is much, much higher (~2 billion). If you simply appended your entire global table (with 20000 rows), along with a column that specified the time, you could do that ~100k times before running in to trouble.

You could also do a hybrid approach, where you have 8000 columns; each time interval would add 3 rows, and the values from the first 8000 rows of your global table would go in the first row of the stats collector. The next 8000 rows of the global table would go in the second row if the stats collector. The next 4000 rows would go in the 3rd row, leaving the last 4k columns in the 3rd row blank. You could use this method to record all 20000 values more than 600k times.

But this question makes me think that there is a better approach possible than any of these suggestions. I don't understand the high-level goal of what you are trying to record, or why. But in 99% of cases with millions of rows or thousands of columns, there's a better way to record the data that you actually need. For example, it seems like most of the 20000 rows in the global table wouldn't change every second. There are ways to make it so that the stats collector only records when one of those values changes, instead of recording all values at every time step. This approach uses a lot less data to capture every value change.