I spent the latter half of yesterday setting up the Hylafax fax server software to send and receive faxes and also accept data calls over its modem using mgetty. There were a few tricks along the way to get it al working right.
Hylafax uses a separate configuration file for for each modem installed and is named after the serial device.
The first setting to get right is GettyArgs. This is where the command is passed through to mgetty. In my case the only argument I passed was %l which represents the modem device.
GettyArgs: " %l"
There were other options that I attempted to pass to mgetty to prevent it from resetting the connection. These were apperently ignored by mgetty which only seemed to look at its own configuration file mgetty.config.
The next interesting command sets up the modem for adaptive answer.
ModemSetupAACmd: AT+FAA=1
With this command in place the modem will attempt to determine if the caller is a fax machine, a modem transmitting data, or a persons voice.
Because I am trying to catch faxes with Hylafax, I need to set
ModemAnswerCmd: AT+FCLASS=1.0;A
to place the modem in the correct fax listening mode when adaptive answer hands off a fax call.
If the call is DATA, it is immediately handed to mgetty which parses its mgetty.config. Mgetty's default is to reset and initialize the modem to answer. Since a connection has already been made by hylafax the following options are necesary to keep the connection alive.
direct y
toggle-dtr n
The final problem I had was with garbage being written to the screen when a DATA call would connect and attempt to log in. This turned out to be a DTR speed mismatch between Hylafax and Mgetty. Set the speed to the same value in both config files and the garbage went away.