DWQA QuestionsCategory: ServerPython scripts can be executed independently, and shell calls are memory-intensive
mywinly asked 5 months ago

Write a python script, the function is to read a 100M file A line by line and write it to another file B after processing. After execution, B will be 600M in size.
It’s okay to execute Python scripts separately, but it’s memory-intensive to call execution with shell scripts
When executed separately, you see it in topSplit.py memory usage < 2%
When the shell call is executed, you see it in topBash memory usage > 20%
Read and write code in Python script:split.py

ReadHandle = open (fileName)

Shell call

/bin/python split.py

Please give me your advice. Thank you.

gaosboy replied 5 months ago

What do you mean by separate execution?

1 Answers
Nonexistent _nemo answered 5 months ago

You write part of the data, such as 100 lines, and flush the contents of the cache into the file.