Well maintained libraries using Popen to interact with a subprocess dynamically?
I work in process controls and have been programming Python for about 10 years now, though it's not my main job function. I have a maintenance utility that was written in C in the mid 90s. It's designed as a session based REPL to upload and download settings (eg PID controller parameters) running as embedded software on controllers that are networked together.
Anyway, I've been working on a library of python code to wrap this utility. This would be very nice, and I could use Python to write some nice maintenance scripts that take full advantage of the programming language compared with something like bash. Most sources say to prefer subprocess.run whenever possible so you avoid deadlock... However, since this is for a session based REPL environment, this is not possible, and I'm being a little more daring by using Popen and reading from the subprocess STDOUT stream in a loop (STDERR redirected to STDOUT).
Fortunately, those who wrote the C program were very smart: every command available will eventually result in a line on STDERR beginning with DONE, FAIL, or FLSH. So I can parse STDERR for those tokens to break out of my readline loop so I don't deadlock my program.
Here's some pseudocode of what I'm doing inside the class:
def __sendCommand(self, cmd: str):
self.__proc.stdin.write(cmd)
self.__proc.stdin.flush()
def __readLines(self):
lines
while True:
line = self.__proc.stdout.readline()
lines.append(line)
if self.getToken(line) in ClassName.TERMINAL_TOKENS:
break
return lines
def exampleCommand(self):
commandText = "LIST -OPTION"
self.__sendCommand(command_text)
stdoutLines = self.__readLines()
result = parseFunc(stdoutLines)
return result
I'm writing dedicated public methods for each command available in the actual utility that parses the stdout to return it in a pythonic way (e.g. LIST will return a dictionary instead of just raw lines text).
Anyway, I have it mostly working, but I would greatly appreciate if anyone knows of any well known / well maintained libraries that interact with a subprocess dynamically like this. I would like to look as some professional examples to compare. Note: anything using subprocess.run(capture_output = True) or Popen.poll() are not examples of this, because they return a completed process.
2 comments · 67% upvoted · other discussions
Maybe pexpect (or wexpect if needed on Windows).
https://pexpect.readthedocs.io/en/stable/
el_extrano [OP] 2 points · Jan 2024
Seems interesting! I had been a bit concerned that my program could deadlock if the child process block-buffers its output.. In testing, I have so far not had that issue. Perhaps the authors manually flushed the buffer? I don't have the source so it's impossible to say.
Apparently that library gets around potential issues by running the child in a pseudo-terminal so it will line buffer its output. Very clever!