Eof: Don't check ftell() in text-mode

ftell() and ftello64() cannot safely be used on text-mode streams in the
way Eof() was using it: It assumed the returned value was a valid file
offset corresponding to the current position, but ftell() isn't guaranteed
to actually return that. It's only guaranteed to return something that can
be used with fseek().

I ran into this issue on Windows when reading a file with LF line endings
(instead of CRLF) in text mode. The ftell() value was different from the
real file offset (probably due to the internal LF => CRLF conversions?),
causing Eof() to return TRUE too early. I've added the test case - it
failed for me on Windows.

It seems that whether ftell() gives a useful result also depends on file
size and the buffer size set with setvbuf(... _IOFBF ...) (FB uses that
too). The point is that ftell() isn't guaranteed to give valid file offsets
in text-mode.

Instead, this patch changes Eof() to peek ahead via getc()/ungetc(), and
check for EOF this way.

Author: dkl

changelog.txt

@@ -18,6 +18,7 @@ Version 1.05.0
 - Compiler crash during error recovery when there was an error when parsing the argument expression for a BYREF AS ANY parameter
 - 1.04.0 regression: Get# for WStrings was incorrectly changed to convert the loaded bytes to wstring characters, like Input# would do. Now it's changed back to just loading the raw bytes into the wstring, which is also how Get# works for other datatypes.
 - 1.04.0 regression: Due to the Get# wstring breakage, the compiler failed to read source files encoded in UTF16LE with BOM on Windows, and UTF32LE with BOM on Linux.
+- Eof() could incorrectly return TRUE too early on Windows, when reading a big text file with LF line endings, OPENed FOR INPUT (text mode)
 
 
 Version 1.04.0

src/rtlib/dev_file_eof.c

@@ -4,7 +4,6 @@
 
 int fb_DevFileEof( FB_FILE *handle )
 {
-	int result;
     FILE *fp;
 
 	FB_LOCK();
@@ -16,32 +15,35 @@ int fb_DevFileEof( FB_FILE *handle )
 		return FB_TRUE;
 	}
 
-	result = FB_FALSE;
-
+	int eof;
 	switch( handle->mode ) {
 	/* non-text mode? */
 	case FB_FILE_MODE_BINARY:
 	case FB_FILE_MODE_RANDOM:
 		/* note: handle->putback_size will be checked by fb_FileEofEx() */
-		result = (ftello( fp ) >= handle->size ? FB_TRUE : FB_FALSE);
+		/* This detects both cases: a) last read reached EOF, b) next
+		   read will reach EOF */
+		eof = (ftello( fp ) >= handle->size);
 		break;
 
 	/* text-mode (INPUT, OUTPUT or APPEND) */
 	default:
-		/* try feof() first, because the EOF char (27) */
-		if( feof( fp ) ) {
-			result = FB_TRUE;
-		/* if in input mode, feof() won't return TRUE if file_pos == file_size */
-		} else if( handle->mode == FB_FILE_MODE_INPUT ) {
-			/* Check the file size, if known
-			   (excluding pipes, for example, which re-use fb_DevFileEof()) */
-			if( handle->hooks->pfnTell != NULL && handle->hooks->pfnSeek != NULL ) {
-				result = (ftello( fp ) >= handle->size ? FB_TRUE : FB_FALSE);
+		/* This also handles the EOF char (27). */
+		/* We can't check ftell(), because it's not guaranteed to give
+		   a real file offset in text mode. */
+		/* a) detect whether last read reached EOF */
+		eof = feof( fp );
+		if( !eof ) {
+			/* b) peek ahead: will the next read reach EOF? */
+			int c = getc( fp );
+			eof = (c == EOF);
+			if( !eof ) {
+				ungetc( c, fp );
 			}
 		}
 		break;
 	}
 
 	FB_UNLOCK();
-	return result;
+	return eof ? FB_TRUE : FB_FALSE;
 }

tests/file/.gitattributes

@@ -0,0 +1 @@
+big-list-eol-lf.txt eol=lf