Response: 500 MLST command not understood [Axway Server]

[anandgmenon]

FTP Server OS: Unix

FTP Server Type: Axway

Client Computer OS: Windows

FluentFTP Version: 49.0.2

Framework: .NET 6

Hey I'm back with another issue/question. So one of our users faced an issue with the GetObjectInfo() method. It seems to be returning
Response: 500 'MLST /CER/INBOX/TEST_CLM.096FB412.20240820161946524': command not understood. See detailed logs below. My understanding from the code is FluentFTP checks if MLST is supported on the server or else fallback to get folder listiting to get the file metadata. Do we have a bug in this logic ?

Logs :

>         Connect(False)
Status:   FluentFTP 49.0.2.0(.NET 6.0)
Status:   Connecting to IP #1= ***:21
Status:   Waiting for a response
Response: 220 10.2.242.199 FTP server 5.5-20220825 ready.     [739117.852d]
Command:  USER ***
Status:   Waiting for response to: USER ***
Response: 331 Password required for ***. [51ms]
Command:  PASS ***
Status:   Waiting for response to: PASS ***
Response: 230 virtual user *** logged in from /52.179.140.75:60198. [69ms]
Command:  FEAT
Status:   Waiting for response to: FEAT
Response: 211-Extensions supportedResponse: AUTHResponse: USERResponse: PASSResponse: QUITResponse: PORTResponse: PASVResponse: EPSVResponse: EPRTResponse: TYPEResponse: STRUResponse: MODEResponse: RETRResponse: STORResponse: ABORResponse: DELEResponse: CWDResponse: XCWDResponse: LISTResponse: NLSTResponse: SITEResponse: SYSTResponse: STATResponse: HELPResponse: NOOPResponse: MKDResponse: XMKDResponse: RMDResponse: XRMDResponse: PWDResponse: XPWDResponse: CDUPResponse: XCUPResponse: SIZEResponse: MDTMResponse: RNFRResponse: RNTOResponse: RESTResponse: FEATResponse: ADATResponse: PROTResponse: PBSZResponse: APPEResponse: XCRCResponse: CCCResponse: COMBResponse: MLSDResponse: UTF8
Response: 211 END [23ms]
Status:   Text encoding: System.Text.UTF8Encoding+UTF8EncodingSealed
Command:  OPTS UTF8 ON
Status:   Waiting for response to: OPTS UTF8 ON
Response: 200 OPTS command successful. [18ms]
Command:  SYST
Status:   Waiting for response to: SYST
Response: 215 UNIX Type: L8 [22ms]
Status:   Active ServerHandler is: None
Status:   Listing parser set to: Machine
Command:  PWD
Status:   Waiting for response to: PWD
Response: 257 "/" is current directory. [18ms]
>         GetObjectInfo("/CER/inbox/TEST_CLM.096FB412.20240820161946524", False)
Command:  MLST /CER/inbox/TEST_CLM.096FB412.20240820161946524
Status:   Waiting for response to: MLST /CER/inbox/TEST_CLM.096FB412.20240820161946524
Response: 500 'MLST /CER/INBOX/TEST_CLM.096FB412.20240820161946524': command not understood. [18ms]
Warning:  Failed to get object info for path /CER/inbox/TEST_CLM.096FB412.20240820161946524 with error 'MLST /CER/INBOX/TEST_CLM.096FB412.20240820161946524': command not understood.

[FanDjango]

My understanding from the code is FluentFTP checks if MLST is supported on the server or else fallback to get folder listiting to get the file metadata.

Correct.

Do we have a bug in this logic ?

I hope not.

But it sure looks like it at first glance.

Here's the what the RFC's say: ( see here )

If the server supports MLST he must advertise that. This then means he supports both, repeat, BOTH MLST and MLSD. The server can only advertise MLST, or not. An advertisement of MLSD is not defined in the RFC.

Non-Standard servers, unfortunately, advertise all sorts of wrong things. Like for example, they advertise MLSD. Per RFC, this doesn't even exist as a valid advertisement. But, in our experience this means that the server supports MLSD and MLST.

This server of yours, it advertises "MLSD" (<-bug #1) and probably wants to say, it supports MLSD but not MLST(<-bug #2).

#1: MLSD is not a RFC allowed FEAT extension advertisement at all
#2: Command MLSD alone (with no MLST comment) is not a RFC allowed extension configuration

So FluentFTP then sets its flag, and MLST is used.

What to do now? Correct FluentFTP's gracious acceptance of (many) servers reporting MLSD and really meaning MLST and MLSD**? Or handle this (one) server, which is also doing something incorrect, by not using MLST?

Note that both are in the wrong. If we change FluentFTP's code to be RFC strictly compliant, both of the miscreants will suffer.

@robinrodricks

The capability should be called MLST in the code and not MLSD. Global rename needed.
The capability is currently correctly set when we see a FEAT response of MLST. This is currently done correctly.
The capability is currently also set when we see a FEAT response of MLSD. <- and this causes the problem.
The problem is: This is actually good for most misbehaving servers, but not good for the one causing this issue.

[robinrodricks]

The capability should be called MLST in the code and not MLSD. Global rename needed.
The capability is currently also set when we see a FEAT response of MLSD. <- and this causes the problem.

I think we should introduce a new capability called MLST in addition to MLSD. Rename or not, I think here the better solution would be:

1. If the server advertises MLST, then we allow both MLST and MLSD commands
2. If the server only advertises MLSD, then we only allow MLSD commands

Is this a workable solution?

FTP server appears to be Axway MFT as per the logs. I have never heard of this server before, and we do not have any FTP server detection logic for this either.

[FanDjango]

If the server only advertises MLSD, then we only allow MLSD commands

I have no inkling how many clients out there rely on MLSD and MLST being allows if only MLSD is advertised, which is the current implementation.

FTP server appears to be Axway MFT as per the logs. I have never heard of this server before, and we do not have any FTP server detection logic for this either.

Wow, where does this appear in the log, I am blind. Edit: Yeah, Google is our friend, ok.

Is this a workable solution?

Well, making MLST and MLSD separate capabilites would make it easier to differentiate. So much is clear. It still leaves the interpretation of only MLSD up to grabs: Either fix this issue and hurt some other users or vice versa. I don't know the numbers and I don't know the facts.

[FanDjango]

The behaviour when "only MLSD" is present: Either we recognize the AxWay Server (which I would prefer, maybe), or we make it configurable by the user.

[robinrodricks]

Either we recognize the AxWay Server (which I would prefer, maybe),

I would also prefer this. I am just a bit unhappy that the server does not specify its name in the welcome message. But this would be the best way because it would "automatically" work for all new users without having to configure or setup anything.

My proposed solution would be:

1. Introduce a new FtpCapability called MLSD (as you have renamed the current one to MLST).
2. The new capability will be synonymous with MLST to preserve current behavior for all current servers
3. We will detect the AxWay server somehow
4. Only for the AxWay server, in its server specific handler, it will set a flag called MLSDIsExclusive = true which indicates that MLSD does not imply MLST
5. By default, for all existing servers, we will use MLSDIsExclusive = false, which means both are synonymous
6. The main implementation of machine listings will use this logic:

// return true if we can use MLST commands
CanUseMLST(){
   if (MLSDIsExclusive){
      return FtpCapability has MLST;
   }else{
      return FtpCapability has MLSD or FtpCapability has MLST;
   }
}

// return true if we can use MLSD commands
CanUseMLSD(){
   if (MLSDIsExclusive){
      return FtpCapability has MLSD;
   }else{
      return FtpCapability has MLSD or FtpCapability has MLST;
   }
}

[FanDjango]

I am just a bit unhappy that the server does not specify its name in the welcome message

Full Ack.

1. and 2. Yes, 4, 5 and 6. Yes

3. We will detect the AxWay server somehow

Hmmm.

10.2.242.199 FTP server 5.5-20220825 ready.

ipaddress "FTP server" n.n-date(yyymmdd) "ready."

How safe would a grep for this be? Risky

[robinrodricks]

ipaddress "FTP server" n.n-date(yyymmdd) "ready."

Its quite specific. It should work.

[anandgmenon]

@FanDjango any updates on this?

[FanDjango]

I don't think anyone has made any progress on this.

[FanDjango]

@anandgmenon Perhaps some incentive for @robinrodricks might be in order to develop special code for a quirky server such as the one you are exhibiting. Patreon? Sponsorship? I dunno, but the custom server mechanism actually also provides for you yourself to (perhaps with some small code changes by us) to cater for this problem.