3

I am doing a lengthy SageMath calculation in Terminal that I plan to save the results of to disk using Shell > Export Text As... The resulting text file will be somewhere in the neighborhood of 10 GB.

Do I need to worry that 16 GB of RAM will not be enough for this task? In particular, I worry that both SageMath and the Terminal display are storing the results separately. SageMath's format would require less space, but by how much I don't know.

Another question: if I try to use “Shell > Export Text As…” while the calculation is running, will this cause it to stop? I'd prefer to not have to start over after three days just in order to find out that the answer to this question is "Yes, it will stop!"

I expect the calculation to take about two weeks total.

Improve this question
5
  • 6
    “I expect the calculation to take about two weeks total.” Wow! Figure out how to redirect that output to a file instead of “Export Text As...” The answer posted suggests this and I say that is truly the best solution to this issue.
    – Giacomo1968
    Jun 1, 2021 at 1:03
  • 1
    The calculation is easy to describe: for each p in Posets(11), print all covering relations. It takes a while to generate 46.75 million inequivalent posets.
    – mathematrucker
    Jun 1, 2021 at 1:55
  • 1
    Since I am going to start over with a standalone script per the answer below, I just did "Shell > Export Text As..." to see what got generated so far. (This did not stop the calculation.) After about 72 hours, 9.5 million posets were generated with a file size of 1.2 GB. If this rate continues the calculation will take about 14 or 15 days and the file shouldn't exceed 8 GB. My previous guesstimate was based on how fast the posets were scrolling in the Terminal window.
    – mathematrucker
    Jun 1, 2021 at 2:08
  • 4
    Well, whatever you are doing it makes more sense to redirect output to a separate file as it progresses. “Export Text As...” is really not well suited for that and the chances of it failing regardless of RAM amount are quote high. Best of luck with this!
    – Giacomo1968
    Jun 1, 2021 at 2:10
  • 3
    Just a tip, add in the ability to restart the script from a given point, because a week of calculations is just asking to be interrupted...
    – Moo
    Jun 1, 2021 at 8:27

1 Answer 1

Reset to default
6

To answer your primary question, the Terminal scroll back buffer size is limited only by your RAM on the machine (16GB). Without knowing what your process is outputting, it is hard to know if this will present an issue or not.

However, rather than depending on the scroll buffer integrity, especially for a process that runs in time measured in weeks, you could instead redirect the output to a file that can be inspected over time.

One way to do this might be to use the logging capabilities of SageMath.

An alternate way might be to create a standalone script, and if you are interested in what would have shown in the terminal (STDOUT), it can be redirected to a file:

your_sage_script > sage.log

Then the data will be redirected to the file sage.log, and can be inspected as that file grows, without affecting the execution of the script itself.

The above ways would be a much better way of ensuring you are saving the I/O of your process. For a script that will take weeks to complete, it is worth looking into these methods.

Improve this answer
2
  • 2
    Thank you for this helpful info. I will write a standalone script.
    – mathematrucker
    Jun 1, 2021 at 1:45
  • 4
    Nice extra benefit of sending the output to a file: It will be faster too! Console output (in Terminal) with scrolling the display up on each new line and appending each line to a HUGE scrollback buffer is a relatively slow action. All that gets bypassed if you redirect the output straight to a file. On small amounts of program output it won't matter much, but in this case it probably saves you hours of overall run-time.
    – Tonny
    Jun 1, 2021 at 7:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .