px_reader can't handle semicolon weirdness in NOTE-field

[jampekka]

File parsing fails with http://www.aluesarjat.fi/database/aluesarjat_kaupunkiverkko/vaesto_sal/vaestoennusteet_sal/b01esps_vaestoennuste.px

The problem is due to it having weird stuff in NOTE-field, in format

NOTE="Some stuff";
"Some stuff more";
"and even more stuff";

from which the reader tries to create new attribute named 'Some stuff more', but fails due to there being non-ascii characters. Fixing would need rewriting the whole metadata tokenizing code.

Seems that the different "statements" are meant to split paragraphs. Don't know if the "spec" (if there is one) allows this ugliness. At least PX-Web seems to eat it: http://www.aluesarjat.fi/graph/Footnote.aspx?File=B01ESPS_Vaestoennuste&path=..%2fDATABASE%2fALUESARJAT_KAUPUNKIVERKKO%2fVAESTO_SAL%2fVAESTOENNUSTEET_SAL%2f&ti=Espoon+v%C3%A4est%C3%B6+1.1.1999-2013+ja+v%C3%A4est%C3%B6ennuste+1.1.2014+-+2023&case=db&ssid=1403051945183&Gedit=false

Fails with both ";-tokenizer" implementations.

[statfi]

Your link gives me only empty note (alaviite). I believe that server is actually Verti, but it should defer to PX Web for parsing PX files anyway.

Spec actually says that you should use # inside NOTE to get line splits.

Sadly getting Python parsing to the level of quality PX Edit has is not that easy. PX the format is relatively easy to get right, but PX the toolkit does not enforce validity. Only solution I know of is using PX Edit for validation and it can also fix easy bugs.

Good parser would go something like this:

• start of file should have letters, not quotation -> this should probably be the first meta key
• find nearest = sign -> you got a meta key between start of file and =
  • split the key by language code [lang] and "subkey"
• language code should be everything between [ and ], no matter how nasty
• subkey can contain just about everything, so first (" and last ") are the only somewhat reliable delimiters
  • try to find " after the key's = sign and the last " immediately followed by ; -> you got the key's value, probably with all the possible nastiness in between safely ignored
  • for safety measure make sure that "; is followed by a valid key as defined above

Then for the next iteration this is repeated until DATA= is met (this should have a straight forward syntax).

This should not be too hard to implement with Python's non-greedy regexp syntax. Pyparsing or similar parser tool should work as well, but I don't have any experience with those...

Question is if there even is use case for complete Python implementation? Since most of the problems in PX files can be already fixed with PX Edit I feel it is a low priority to fix the harder parsing bugs. If bugs are met in table structure parsing that would another story.

Perhaps erroneous keys and values could just be skipped and set a flag to alert the user about possible errors? This should be just a simple try/except block.