Skip to content

Commit 9f7279c

Browse files
s-m-es-m-e
committed
0.0.3 release
2 parents d91e346 + 3522415 commit 9f7279c

File tree

18 files changed

+1184
-316
lines changed

18 files changed

+1184
-316
lines changed

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ nosetests.xml
4646
coverage.xml
4747
*,cover
4848
.hypothesis/
49+
.pytest_cache/
4950

5051
# Translations
5152
*.mo
@@ -97,3 +98,5 @@ test_mount
9798
test_logs
9899
fsx-linux*
99100
screenshots/
101+
notebooks/
102+
notes.md

‎CHANGES.rst‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,19 @@
11
Changes
22
=======
33

4+
0.0.3 (2019-05-01)
5+
------------------
6+
7+
* FEATURE: LoggedFS-python can be used as a library in other Python software, enabling a user to specify callback functions on filesystem events. The relevant infrastructure is exported as ``loggedfs.loggedfs_notify``. See library example under ``docs``.
8+
* FEATURE: New programmable filter pipeline, see ``loggedfs.filter_field_class``, ``loggedfs.filter_item_class`` and ``loggedfs.filter_pipeline_class``
9+
* FEATURE: New flag ``-b``, explicitly activating logging of read and write buffers
10+
* FEATURE: In "traditional" logging mode (not JSON), read and write buffers are also logged zlib-compressed and BASE64 encoded.
11+
* FEATURE: Convenience function for decoding logged buffers, see ``loggedfs.decode_buffer``
12+
* FIX: LoggedFS-python would have crashed if no XML configuration file had been specified.
13+
* FIX: **Directory listing (``ls``) was broken.**
14+
* FIX: Testing infrastructure did not catch all exceptions in tests.
15+
* FIX: Testing infrastructure did not handle timeouts on individual tests correctly.
16+
417
0.0.2 (2019-04-23)
518
------------------
619

‎docs/library_example.rst‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
Using LoggedFS-python as a library
2+
==================================
3+
4+
Create a new directory, for instance in your current working directory, named ``demo_dir``. Then fire up an interactive Python shell such as Jupyter Notebook or Jupyter Lab. Now you can try the following:
5+
6+
.. code:: python
7+
8+
import re
9+
import loggedfs
10+
11+
demo_filter = loggedfs.filter_pipeline_class(
12+
include_list = [loggedfs.filter_item_class([loggedfs.filter_field_class(
13+
name = 'proc_cmd', value = re.compile('.*kate.*').match
14+
)])]
15+
)
16+
demo_data = []
17+
demo_err = []
18+
19+
demo = loggedfs.loggedfs_notify(
20+
'demo_dir',
21+
background = True,
22+
log_filter = demo_filter,
23+
consumer_out_func = demo_data.append,
24+
consumer_err_func = demo_err.append
25+
)
26+
27+
You have just stated recording all filesystem events that involve a command containing the string ``kate``. Leave the Python shell and write some stuff into the ``demo_dir`` using ``Kate``, the KDE text editor. Once you are finished, go back to your Python shell and terminate the recording.
28+
29+
.. code:: python
30+
31+
demo.terminate()
32+
33+
Notice that the recorded data ends with an "end of transmission" marker. For convenience, remove it first:
34+
35+
.. code:: python
36+
37+
assert isinstance(demo_data[-1], loggedfs.end_of_transmission)
38+
demo_data = demo_data[:-1]
39+
40+
Let's have a look at what you have recorded:
41+
42+
.. code:: python
43+
44+
print(demo_data[44]) # index 44 might show something different in your case
45+
46+
::
47+
48+
{'proc_cmd': '/usr/bin/kate -b /test/demo_dir/demo_file.txt',
49+
'proc_uid': 1000,
50+
'proc_uid_name': 'ernst',
51+
'proc_gid': 100,
52+
'proc_gid_name': 'users',
53+
'proc_pid': 11716,
54+
'action': 'read',
55+
'status': True,
56+
'param_path': '/test/demo_dir/demo_file.txt',
57+
'param_length': 4096,
58+
'param_offset': 0,
59+
'param_fip': 5,
60+
'return_len': 1486,
61+
'return': '',
62+
'time': 1556562162704772619}
63+
64+
Every single event is represented as a dictionary. ``demo_data`` is therefore a list of dictionaries. The following columns / keys are always present:
65+
66+
- proc_cmd: Command line of the process ordering the operation.
67+
- proc_uid: UID (user ID) of the owner of the process ordering the operation.
68+
- proc_uid_name: User name of the owner of the process ordering the operation.
69+
- proc_gid: GID (group ID) of the owner of the process ordering the operation.
70+
- proc_gid_name: Group name of the owner of the process ordering the operation.
71+
- proc_pid: PID (process ID) of the process ordering the operation.
72+
- action: Name of filesystem operation, such as ``open``, ``read`` or ``write``.
73+
- status: Boolean, describing the success of the operation.
74+
- return: Return value of operation. ``None`` if there is none.
75+
- time: System time, nanoseconds, UTC
76+
77+
Other columns / keys are optional and depend on the operation and its status. With this knowledge, you can run typical Python data analysis frameworks across this data. Pandas for instance:
78+
79+
.. code:: python
80+
81+
import pandas as pd
82+
data_df = pd.DataFrame.from_records(demo_data, index = 'time')
83+
84+
data_df[data_df['action'] == 'write'][['param_buf_len', 'param_offset', 'return']]
85+
86+
::
87+
88+
param_buf_len param_offset return
89+
time
90+
1556562164301499774 57.0 0.0 57
91+
1556562164304043463 2.0 57.0 2
92+
1556562164621417400 1487.0 0.0 1487
93+
1556562165260276486 53.0 0.0 53
94+
1556562165532797611 1486.0 0.0 1486

‎setup.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@
4343

4444

4545
# Bump version HERE!
46-
_version_ = '0.0.2'
46+
_version_ = '0.0.3'
4747

4848

4949
# List all versions of Python which are supported
@@ -99,7 +99,7 @@
9999
keywords = ['filesystem', 'fuse', 'logging', 'monitoring'],
100100
include_package_data = True,
101101
install_requires = [
102-
'click',
102+
'click>=7.0',
103103
'fusepy @ git+https://github.com/s-m-e/fusepy@master#egg=fusepy-2.0.99',
104104
'xmltodict'
105105
],

‎src/loggedfs/__init__.py‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,16 @@
2929
# IMPORT
3030
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
3131

32-
from .cli import cli_entry
33-
from .core import (
34-
loggedfs,
32+
from ._core.cli import cli_entry
33+
from ._core.filter import (
34+
filter_field_class,
35+
filter_item_class,
36+
filter_pipeline_class
37+
)
38+
from ._core.fs import (
39+
_loggedfs,
3540
loggedfs_factory
3641
)
42+
from ._core.ipc import end_of_transmission
43+
from ._core.notify import notify_class as loggedfs_notify
44+
from ._core.out import decode_buffer

‎src/loggedfs/_core/__init__.py‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# -*- coding: utf-8 -*-
2+
3+
"""
4+
5+
LoggedFS-python
6+
Filesystem monitoring with Fuse and Python
7+
https://github.com/pleiszenburg/loggedfs-python
8+
9+
src/loggedfs/_core/__init__.py: Module core init
10+
11+
Copyright (C) 2017-2019 Sebastian M. Ernst <[email protected]>
12+
13+
<LICENSE_BLOCK>
14+
The contents of this file are subject to the Apache License
15+
Version 2 ("License"). You may not use this file except in
16+
compliance with the License. You may obtain a copy of the License at
17+
https://www.apache.org/licenses/LICENSE-2.0
18+
https://github.com/pleiszenburg/loggedfs-python/blob/master/LICENSE
19+
20+
Software distributed under the License is distributed on an "AS IS" basis,
21+
WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the
22+
specific language governing rights and limitations under the License.
23+
</LICENSE_BLOCK>
24+
25+
"""

‎src/loggedfs/cli.py‎ renamed to ‎src/loggedfs/_core/cli.py‎

Lines changed: 42 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
Filesystem monitoring with Fuse and Python
77
https://github.com/pleiszenburg/loggedfs-python
88
9-
src/loggedfs/cli.py: Command line interface
9+
src/loggedfs/_core/cli.py: Command line interface
1010
1111
Copyright (C) 2017-2019 Sebastian M. Ernst <[email protected]>
1212
@@ -31,8 +31,9 @@
3131

3232
import click
3333

34-
from .core import loggedfs_factory
35-
from .filter import parse_filters
34+
from .defaults import LOG_ENABLED_DEFAULT, LOG_PRINTPROCESSNAME_DEFAULT
35+
from .fs import loggedfs_factory
36+
from .filter import filter_pipeline_class
3637

3738

3839
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
@@ -70,54 +71,70 @@
7071
is_flag = True,
7172
help = 'Format output as JSON instead of traditional loggedfs format.'
7273
)
74+
@click.option(
75+
'-b', '--buffers',
76+
is_flag = True,
77+
help = 'Include read/write-buffers (compressed, BASE64) in log.'
78+
)
79+
@click.option(
80+
'--lib',
81+
is_flag = True,
82+
help = 'Run in library mode. DO NOT USE THIS FROM THE COMMAND LINE!',
83+
hidden = True
84+
)
7385
@click.argument(
7486
'directory',
7587
type = click.Path(exists = True, file_okay = False, dir_okay = True, resolve_path = True)
7688
)
77-
def cli_entry(f, p, c, s, l, json, directory):
89+
def cli_entry(f, p, c, s, l, json, buffers, lib, directory):
7890
"""LoggedFS-python is a transparent fuse-filesystem which allows to log
79-
every operations that happens in the backend filesystem. Logs can be written
80-
to syslog, to a file, or to the standard output. LoggedFS comes with an XML
91+
every operation that happens in the backend filesystem. Logs can be written
92+
to syslog, to a file, or to the standard output. LoggedFS-python allows to specify an XML
8193
configuration file in which you can choose exactly what you want to log and
8294
what you don't want to log. You can add filters on users, operations (open,
83-
read, write, chown, chmod, etc.), filenames and return code. Filename
84-
filters are regular expressions.
95+
read, write, chown, chmod, etc.), filenames, commands and return code.
8596
"""
8697

8798
loggedfs_factory(
8899
directory,
89-
**__process_config__(c, l, s, f, p, json)
100+
**__process_config__(c, l, s, f, p, json, buffers, lib)
90101
)
91102

92103

93104
def __process_config__(
94105
config_fh,
95106
log_file,
96107
log_syslog_off,
97-
fuse_foreground_bool,
98-
fuse_allowother_bool,
99-
log_json
108+
fuse_foreground,
109+
fuse_allowother,
110+
log_json,
111+
log_buffers,
112+
lib_mode
100113
):
101114

102115
if config_fh is not None:
103-
config_xml_str = config_fh.read()
116+
config_data = config_fh.read()
104117
config_fh.close()
118+
(
119+
log_enabled, log_printprocessname, filter_obj
120+
) = filter_pipeline_class.from_xmlstring(config_data)
105121
config_file = config_fh.name
106122
else:
107-
config_file = '[None]'
108-
config_xml_str = None
109-
110-
config_dict = parse_filters(config_xml_str)
123+
log_enabled = LOG_ENABLED_DEFAULT
124+
log_printprocessname = LOG_PRINTPROCESSNAME_DEFAULT
125+
filter_obj = filter_pipeline_class()
126+
config_file = None
111127

112128
return {
113-
'log_includes': config_dict['log_includes'],
114-
'log_excludes': config_dict['log_excludes'],
115-
'log_enabled': config_dict['log_enabled'],
116-
'log_printprocessname': config_dict['log_printprocessname'],
129+
'fuse_foreground': fuse_foreground,
130+
'fuse_allowother': fuse_allowother,
131+
'lib_mode': lib_mode,
132+
'log_buffers': log_buffers,
133+
'_log_configfile' : config_file,
134+
'log_enabled': log_enabled,
117135
'log_file': log_file,
118-
'log_syslog': not log_syslog_off,
119-
'log_configmsg': 'LoggedFS-python using configuration file %s' % config_file,
136+
'log_filter': filter_obj,
120137
'log_json': log_json,
121-
'fuse_foreground_bool': fuse_foreground_bool,
122-
'fuse_allowother_bool': fuse_allowother_bool
138+
'log_printprocessname': log_printprocessname,
139+
'log_syslog': not log_syslog_off
123140
}

‎src/loggedfs/_core/defaults.py‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# -*- coding: utf-8 -*-
2+
3+
"""
4+
5+
LoggedFS-python
6+
Filesystem monitoring with Fuse and Python
7+
https://github.com/pleiszenburg/loggedfs-python
8+
9+
src/loggedfs/_core/defaults.py: Default configurations
10+
11+
Copyright (C) 2017-2019 Sebastian M. Ernst <[email protected]>
12+
13+
<LICENSE_BLOCK>
14+
The contents of this file are subject to the Apache License
15+
Version 2 ("License"). You may not use this file except in
16+
compliance with the License. You may obtain a copy of the License at
17+
https://www.apache.org/licenses/LICENSE-2.0
18+
https://github.com/pleiszenburg/loggedfs-python/blob/master/LICENSE
19+
20+
Software distributed under the License is distributed on an "AS IS" basis,
21+
WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License for the
22+
specific language governing rights and limitations under the License.
23+
</LICENSE_BLOCK>
24+
25+
"""
26+
27+
28+
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
29+
# CONST
30+
# +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
31+
32+
FUSE_ALLOWOTHER_DEFAULT = False
33+
FUSE_FOREGROUND_DEFAULT = False
34+
35+
LIB_MODE_DEFAULT = False
36+
37+
LOG_BUFFERS_DEFAULT = False
38+
LOG_ENABLED_DEFAULT = True
39+
LOG_JSON_DEFAULT = False
40+
LOG_PRINTPROCESSNAME_DEFAULT = True
41+
LOG_SYSLOG_DEFAULT = False

0 commit comments

Comments
 (0)