Skip to content

Commit ca0927c

Browse files
spirillenNissar Chababy
andauthored
Documente how to use ramfs for faster I/O = faster test (#252)
Contributors: * @spirillen * Knowen Issue Cygwin Have made some notes about the limitation of running Cygwin or Python on a Windows and how it is affecting @PyFunceble Signed by @spirillen * Added Power(less)shell to the title for know Windows issues... Signed by @spirillen * Major changes to docs/ With this release I have made some minor textual changes, but big changes to the filestructure as I have renamed the files inside docs/usage/. This is done to make the filename + the titles more systematic and equal, it just leaves the "mess" it used to be. Signed by @spirillen * Tiny fixup of the text for .PyFunceble.yaml Ensuring that no misunderstands the .PyFunceble.yaml is overwritten by default since .PyFunceble.overwrite.yaml was introduced. * Updated example IP's to RFC:5737 Signed by @spirillen * Ordering the Usage docs Fixed a small Wording Refs: #247 (comment) Signed by @spirillen * Experimental filenaming for deprecated args Relates to #247 (comment) fix #247 (comment) Signed by @spirillen * Rename bash-terminal.rst to terminal.rst Fixes #247 (comment) * Update index.rst in docs Forgot to update the docs/usage/idex.rst to meet the new filename of terminal.rst. * Small rewrite of the intro in terminal.rst And a few missing duble lines between arguments Previous commit within same PR tuched https://mypdns.org/spirillen/PyFunceble/-/issues/26 * Wrote in depth about .PyFunceble.overwrite.yaml I also spotted a few lines, which was pushing the 72 chars per line boundries. (72 vs 80 chars per line, for support of Unix consoles) This commit tuches #246 to better desription of the .PyFunceble.overwrite.yaml as requested by @ZeroDot1 * Fixing a BUG in the code example By a mistake there was written `-url "$URI"` which of curse should have been `-u "$URI"`. The mistake was spotted in easylist/easylist#9061 (comment) posted by @ryanbr * Written about how to speedup test with ramFS Solved a few github link issues to @spirillen's profile * Improvement of ramFS text Have fixed markup of code-block fixed markup of the indented paragraph and changed it to a info block, as it seems more right Fixed the Url's * Trying to fix typo and grammar. Co-authored-by: Nissar Chababy <[email protected]> Co-authored-by: Spirillen Marsupilami <[email protected]>
1 parent 7c9a3f8 commit ca0927c

File tree

3 files changed

+93
-15
lines changed

3 files changed

+93
-15
lines changed

docs/facts/faq.rst

Lines changed: 90 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -5,15 +5,17 @@ How to speed up the test process?
55
---------------------------------
66

77
.. warning::
8-
Beware, when talking about speed a lot a thing have to be taken in consideration.
9-
Indeed here is a non exhaustive list of things which fluctuate testing speed.
8+
Beware, when talking about speed a lot of things have to be taken into consideration.
9+
Indeed here is a non-exhaustive list of things that fluctuate the testing speed.
1010

1111
* Bandwidth.
1212
* DNS Server response time.
1313
* CPU.
14-
* ISP blocking a big amount of connection to the outside world.
14+
* ISP's who blocks a big amount of connection to the outside world.
1515
* Our databases management (do not apply for MySQL and MariaDB format).
1616
* Amount of data to test.
17+
* Disk I/O in particular as PyFunceble is heavy on the I/O
18+
* RamDrives and NVME disks are very suitable for PyFunceble CSV db.
1719
* ...
1820

1921
I have a dedicated server or machine just for PyFunceble
@@ -22,21 +24,97 @@ I have a dedicated server or machine just for PyFunceble
2224
Simply increase the number of maximal workers PyFunceble is allowed to use
2325
through the `--max-workers <../usage/index.html#w-max-workers>`_ argument.
2426

25-
By default the number of workers is equal to:
27+
By default, the number of workers is equal to:
2628

2729
::
2830

2931
CPU CORES - 2
3032

31-
meaning that if you have 8 CPU cores, the value will be automatically set to
32-
:code:`6`.
33+
meaning that if you have :code:`8` CPU threads, the value will be
34+
automatically set to :code:`6`.
3335

3436

3537
.. warning::
36-
Keep in mind that the :code:`--max-workers` mostly - if not only - affects
37-
the tester processes. Because we want to safely write the files, we still
38-
need a single processes which read the submitted results and generate the
39-
outputs.
38+
Keep in mind that the :code:`--max-workers` (:code:`-w`) mostly - if
39+
not only - affects the tester processes. Because we want to safely
40+
write the files(Disk I/O), we still need a single process that reads the
41+
submitted results and generates the outputs.
4042

41-
The reason we added this to PyFunceble :code:`4.0.0` is we don't want to
42-
have a wrongly formatted output file.
43+
The reason we added this to PyFunceble :code:`4.0.0` is we don't want
44+
to have a wrongly formatted file output.
45+
46+
47+
Setup and use ramfs
48+
-------------------
49+
What is a ramfs and why not use tmpfs?
50+
51+
:code:`ramfs` is better than :code:`tmpfs` when data needs to be secret,
52+
since :code:`ramfs` data never gets swapped (saved to a physical storage
53+
drive), while tmpfs may get swapped.
54+
Third parties who later gain root or physical access to the machine then
55+
can inspect the swap space and extract sensitive data.
56+
57+
The HOWTO solution
58+
^^^^^^^^^^^^^^^^^^
59+
You can prepare :code:`ramfs` mount so any non-privileged user can
60+
mount/unmount it on-demand.
61+
62+
To do this, you will need root privilege, once. Ask the administrator of
63+
your system to set this up for you, if you lack root privileges.
64+
65+
At first, you need to add a line to the :code:`/etc/fstab`. The line in
66+
fstab may look like this:
67+
68+
69+
.. :code-block:: console
70+
none /mnt/ramfs ramfs noauto,user,size=1024M,mode=0777 0 0
71+
72+
* :code:`/mnt/ramfs` is a mount point, where the ramfs filesystem will be
73+
mounted. Directory **most** exist.
74+
* :code:`noauto` option prevents this from being mounted automatically
75+
(e.g. at system's boot-up).
76+
* :code:`user` makes this mountable by regular users.
77+
* :code:`size` sets this "ramdisk's" size (you can use :code:`M` and
78+
:code:`G` here)
79+
* :code:`mode` is very important, with the octal :code 0770 only root and
80+
user, who mounted this filesystem, will be able to read and write to
81+
it, not the others (you may use different :code of your choice as well,
82+
but be very sure about it!).
83+
84+
.. note::
85+
86+
We recommend you to set the file mode to :code:`0777` in case you
87+
are using this in relation to any kind of scripting, to ensure
88+
subprocesses have access to the file(s). In all other cases, you should set the folder
89+
permision to :code:`0770`.
90+
91+
**Mount**
92+
93+
.. code-block:: console
94+
95+
$ mount /mnt/ramfs/
96+
97+
98+
**Unmount**
99+
100+
.. code-block:: console
101+
102+
$ umount /mnt/ramfs/
103+
104+
105+
This chapter has practically been copied from
106+
`<https://unix.stackexchange.com/a/325421>`_ creditted to Neurotransmitter
107+
as it is well written and cover our purpose for describing how to setup a
108+
ramFS to be used for testing with PyFunceble.
109+
110+
Next, we need to configure PyFunceble to use the newly created and mounted
111+
ramFS. This is done with the
112+
`PYFUNCEBLE_OUTPUT_LOCATION <../usage/index.html#global-variables>`_ ; now
113+
all outputs are stored in the ramFS, so remember to copy the results to a
114+
stationary file path when you are done.
115+
116+
Next time you are going to run a test with PyFunceble are will do:
117+
1. Mount the ramFS
118+
2. Copy the last test results to the ramFS
119+
3. Run your test
120+
4. Copy your results from ramFS to a stationary file path.

docs/special-thanks.rst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,8 @@ which helped and/or still help me build, test and or make `PyFunceble`_ a better
7575
.. _@rthalley: https://github.com/rthalley
7676
.. _@ScriptTiger: https://github.com/ScriptTiger
7777
.. _@SMed79: https://github.com/SMed79
78-
.. _@spirillen: https://github.com/spirillen
78+
.. _@spirillen: https://mypdns.org/spirillen
7979
.. _@tartley: https://github.com/tartley
8080
.. _@theskumar: https://github.com/theskumar
8181
.. _@yaml: https://github.com/yaml
82-
.. _@ybreza: https://github.com/ybreza
82+
.. _@ybreza: https://github.com/ybreza

docs/usage/terminal.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1072,7 +1072,7 @@ CPU - 2. Otherwise, it will 1.
10721072
- **This means you should be experimenting a bit your self.**
10731073

10741074
To follow the "behind the scene" talk about the subject, please take a
1075-
look at `issue <https://github.com/spirillen/PyFunceble/issues/34>`_
1075+
look at `issue <https://mypdns.org/spirillen/PyFunceble/-/issues/34>`_
10761076

10771077

10781078
------

0 commit comments

Comments
 (0)