Always Be Saving

[nickzoic]

Working on datasets in Countess I find myself adding "Save CSV" plugins scattered all over the place, to store intermediate results, which obscures the flow of data quite a lot.

For example, compare these:

Before	After

Maybe it'd be more sensible to have an automatic "Save As" at the bottom of every node to reduce clutter?

Plus (maybe) saving intermediate results would be an efficient way to handle regeneration of data?

[afrubin]

Yeah I think this is a good idea to add an "Output to CSV" checkbox for every node.

Rather than "Save As" that implies the user should provide their own file path/file name, we should automatically generate this file name and put it in a csv/ directory under the associated working directory for the pipeline (that is set globally for the whole run). The automatic file names would start with the datetime and then have a mutated version of the plugin name (kebab case? underscores?). Note that this may constrain what characters are allowed to exist in the plugin names, or certain invalid characters could just be stripped out.

[nickzoic]

Hmmm, interesting, any step which saves as CSV can of course be retrieved as CSV which could be used to restart partial computation but I wonder if we'd be better off saving these intermediate files as something more flexible & efficient like parquet, or just persisting data to a DuckDB database.

[afrubin]

Hmmm, interesting, any step which saves as CSV can of course be retrieved as CSV which could be used to restart partial computation but I wonder if we'd be better off saving these intermediate files as something more flexible & efficient like parquet, or just persisting data to a DuckDB database.

I see these as serving two very different purposes. The CSV is for people who want to inspect/plot an intermediate artefact, and parquet is for resuming workflows (which I agree is a good thing to have). So maybe the answer is both?