Separates AppendText for MessagesLog. Adds process memory limit on Me… (#2086)

HelioGuilherme66 · Helio Guilherme · Nyral · web-flow · commit ec7ea5703664 · 2020-06-29T01:19:17.000+01:00
* Separates AppendText for MessagesLog. Adds process memory limit on MessagesLog. * Adds psutil to requirements. * Closes and reopens Messages Log window when process memory exceeds 80% * Key shortcuts update. * Fixes update notification * Improves Help links and About * Version 1.7.4rc1 * Update ride-1.7.4.rst Corrected rc1 version * Update ride-1.7.4.rst * Release notes for 1.7.4rc1 * Update ride-1.7.4.rst Make final version. * Development version * Improves release notes builder * Update ride-1.7.4.rst * Changes log to use unicode when on Python 2.7 (#2091) * Moved first condition at the end of the key processing event * Refactored structure of conditions in `OnKeyDown` * Makes some unit test files runnable * Version 1.7.4 * Back to development of 1.7.4.1 * Fixed shortcut in keyword tooltip. We no longer use Ctrl-M. (#2099) * Fixed shortcut in keyword tooltip. We no longer use Ctrl-M. * Updated implementation to use localize_shortcuts. * Fixed SelectAll issue in Grid Editor (#2096) * Moved first condition at the end of the key processing event * Refactored structure of conditions in `OnKeyDown` * Added debug prints and SelectAll shortcut when pressing Alt+A * Revert "Fixed SelectAll issue in Grid Editor (#2096)" (#2100) This reverts commit b7336e0. * Fixes newlines in cell tooltip (#2101) * Fixes newlines in cell tooltip * Fix tooltip unit test * Updates Maven and seleniumlibrary versions. (#2103) * Makes Text Editor read-only if file is read-only. Visual hint is back… (#2102) * Makes Text Editor read-only if file is read-only. Visual hint is background color change. * Updates Maven and seleniumlibrary versions. (#2103) * Fix named colors and update when file is read-only with focus on Text Editor * Update Travis to ubuntu xenial, wxPython and python 3.7 (#2104) * Corrects text update when on read-only on Text Editor (#2106) Fix read file when on read only on Text Editor * Fix for #1421, Repeated Resource files on Tree when on Windows (#2105) * Fix #2107 - Avoids changes in Resource file name when content assist. (#2109) * Removes changes to resource file name for suggestions of keywords * Avoids RuntimeError with ContentAssist for Resource Files * Fixes one test and reactivates other * Updates robotframework-mavenplugin version (#2110) * Uses unicode to calculate font name size when on Python 2.7 (#2112) * Made shortcuts from RIDE and Parser Log tabs to unregister when they're not selected * Removed debug prints and made Select All shortcut work (wasn't implemented) for Ride and Parser Log * Reverted workaround in previous commits and replaced `register_shortcuts` with OnKeyDown events to prevent further conflicts * Misc fixes (#2116) * Corrects Mac shortcut keys * Corrects messages for wxPython versions. * Added back shortcut registration for Cmd-C * Fixed SelectAll issue in Grid Editor, RIDE Log and Parser Log (#2113) * Moved first condition at the end of the key processing event * Refactored structure of conditions in `OnKeyDown` * Made shortcuts from RIDE and Parser Log tabs to unregister when they're not selected * Removed debug prints and made Select All shortcut work (wasn't implemented) for Ride and Parser Log * Reverted workaround in previous commits and replaced `register_shortcuts` with OnKeyDown events to prevent further conflicts * Added back shortcut registration for Cmd-C * Fixed SelectAll issue in Grid Editor, RIDE Log and Parser Log (#2113) * Moved first condition at the end of the key processing event * Refactored structure of conditions in `OnKeyDown` * Made shortcuts from RIDE and Parser Log tabs to unregister when they're not selected * Removed debug prints and made Select All shortcut work (wasn't implemented) for Ride and Parser Log * Reverted workaround in previous commits and replaced `register_shortcuts` with OnKeyDown events to prevent further conflicts * Added back shortcut registration for Cmd-C * Fixed SelectAll issue in Grid Editor, RIDE Log and Parser Log (#2113) * Moved first condition at the end of the key processing event * Refactored structure of conditions in `OnKeyDown` * Made shortcuts from RIDE and Parser Log tabs to unregister when they're not selected * Removed debug prints and made Select All shortcut work (wasn't implemented) for Ride and Parser Log * Reverted workaround in previous commits and replaced `register_shortcuts` with OnKeyDown events to prevent further conflicts * Added back shortcut registration for Cmd-C * Version 1.7.4.1 * Update ride-1.7.4.rst removed mentions to last support to python 2.7 * Update README.rest Version 1.7.4.1. * Back to development version - 2.0 * Replaces python 2.7 by 3.6 * Format Readme * Cleans Invoke * Removes python 2.7 from test_all.sh * Cleans tasks.py * Cleanup of wxPython/wxPhoenix version conditioning (#2121) * Adds pycodestyle for PyCharm PEP8 analysis * Removal of wxPython version conditions, code cleanup phase 1 * Removal of wxPython version conditions, code cleanup phase 1 * Removal of wxPython version conditions, code cleanup phase 1 * Removal of wxPython version conditions, code cleanup phase 1 * Removal of wxPython version conditions, code cleanup phase 1 * Removal of wxPython version conditions, code cleanup phase 1 * Change max line length to 90 char, for PyCharm * Removes Python 2.7 support (#2124) * Adds pycodestyle for PyCharm PEP8 analysis * Removal of wxPython version conditions, code cleanup phase 1 * Removal of wxPython version conditions, code cleanup phase 1 * Removal of wxPython version conditions, code cleanup phase 1 * Removal of wxPython version conditions, code cleanup phase 1 * Removal of wxPython version conditions, code cleanup phase 1 * Removal of wxPython version conditions, code cleanup phase 1 * Change max line length to 90 char, for PyCharm * Main removal of PY2 and PY3 * Final removal of PY2 and PY3 and basestring, unicode and unichar functions. * wx.NewId() to wx.NewIdRef() (#2125) * Changes wx.NewId to wx.NewIdRef * Add AppVeyor configuration (#2131) * Adds an autoclose timer to the Create Shortcut dialog. (#2130) * Adds an autoclose timer to the Create Shotcut dialog. * Changes Travis to use Ubuntu 18.04 * Appveyor (#2132) * Add AppVeyor configuration * Adds tox config * Changes install to be with tox (#2133) * Fixed Python 3.8 incompatibility (#2128) * Fixed setlocale error * Updated configobj to version 5.0.6 (latest). Old version was causing issues when reading from settings.cfg * Fixed small issue in preferences. Zoom factor wasn't getting set to negative values. * Fixes unit tests for #2128 (#2137) * Fixed setlocale error * Updated configobj to version 5.0.6 (latest). Old version was causing issues when reading from settings.cfg * Fixed small issue in preferences. Zoom factor wasn't getting set to negative values. * Fixes triggered errors and unit tests Co-authored-by: Nyral <39315668+Nyral@users.noreply.github.com> * Fix travis (#2138) * Removes installation of RIDE. Avoids Segmentation fault on tags test * Fixes bad character when pressing a first key on Grid Editor * Fixes reprocessing of %date% %time% variables on Windows (#2141) * Inform that development branch is compatible with Python 3.8 * Cleans up unittests (#2142) * Fix one error that was being ignored * Restores two skipped tests. * Fixes resources and file unit tests * Better reason for skipped test. Removes errors of unclosed files * Reduced to two skipped tests * Attempt to clean C++ exceptions, only happens with Invoke * Fixed focus issue when using F2 to edit grid cell (#2140) * Added shortcut handling for F2 in order to set focus * Reverted previous commit and removed StartingClick because it's redundant. Text selection breaks if SelectAll() or SetSelection() is called after SetFocus(). This commit fixes that issue for double click and F2. * Added white color to inside the icon face (#2154) * Fixed crash when you delete tag and more (#2155) * Added condition to prevent event from return key from crashing RIDE * Added fix for when tag box remains empty when you delete the text you were writing in it; Added fix for TagBox deleted error when KillFocus is called. * Turns Test Suite Tree and File Explorer into Plugins (#2152) * Fix one error that was being ignored * First experiments * Test Suite Tree panel is now a Plugin * Changes File Explorer to be a Plugin * Added white color to inside the icon face (#2154) * Fixed crash when you delete tag and more (#2155) * Added condition to prevent event from return key from crashing RIDE * Added fix for when tag box remains empty when you delete the text you were writing in it; Added fix for TagBox deleted error when KillFocus is called. * Fixes perspectives on Tree pane * Fixes Perspectives and changes File Explorer to be a Plugin Co-authored-by: Nyral <39315668+Nyral@users.noreply.github.com> * Fixes default arguments keyword help, #2134 (#2157) * Adds requirement for Python >=3.6 * Removes Python version conditioning * Adds APPDATA to environment and build step with Invoke. * Adds APPDATA var to tox.ini, changes to tox test * Adds badges for Travis and AppVeyor unit tests status * Adds README with ASCIIDoc format, will replace exisiting * Encloses text in Cell or selected text with certain symbols (#2158) * Encloses text in Cell or selected text with certain symbols * Corrects position of cursor. Fixes text selection highlight * Fixes select text (#2160) * Adds a flag to wx.TextControl to force selection highlight on Windows. * Text Editor - Encloses text in Cell or selected text with certain symbols. Adds support for Ctrl-3 and Ctrl-4 (Comment/Uncomment) * Fix comment/uncomment index range in Text Editor * Fix enclosing text * Fix variables creation * Fix keys in Grid editor * Fix keys in Grid Editor, [ and { still don't trigger * Fix error message from TreePlugin * Fix list selection on contextsuggestion. BREAKS enclose text in cell editor * Disable enclose text in cell editor when in Linux * Fixes { and [ in grid editor. Still disable in cell editor, on Linux * Adds autoident to Text Editor * Auto-indents, when first Task, Test or Keyword * Fixes { and [ in grid editor. Still disable in cell editor, on Linux * Fixes { and [ in Windows, grid editor * Fix keys (#2178) * Fixes { and [ in grid editor. Still disable in cell editor, on Linux * Avoids beep on keywords suggestions when on Linux * Adds ! to Save item on menu. Not detected when removed. (#2180) * Indents and de-indents block of selected text (#2181) * Experiments getting selection lines * Indent and de-indent blocks of text * Correct imports (#2182) * Adds conditions for wxPython 4.1.0 (#2184) * Upgrade wxPython to 4.1.0 (#2187) * Adds conditions for wxPython 4.1.0 * Fixes #2186 * Update docs about v1.7.4.2 * Log misc fixes (#2198) * Fixes syntax error * Makes error in removing log files a byte buffer. * Makes a test compatible with RF 3.2.1 * Better error and removal of old log files. (#2199) * Changes code to allow starting with File Explorer or Tree plugins disabled * Fix parsing warning of report/ log html file (#2212) Should not show this warning message if there're report/log html file in the workspace. Co-authored-by: Johnny Huang <jnhuang@telenavsoftware.com> * Fix for 1668: Given a large test suite, selecting all tests from the root of tree, freezes RIDE for some time (#2210) * 1668: The Select All Tests command on the root of a deeply nested test folder, freezes RIDE for several minutes. The problem is caused by Tree.SelectAllTests strategy: expanding and collapsing all the nodes. This causes CPU waste to expand/collapse (and maybe render) and wait for the creation of the tree node handlers. The idea of the fix is to navigate the tests using the link between controllers. In order to avoid all the dancing of the tree nodes. In this commit: - Changed Tree.SelectAllTests() to extract all the TestCaseController(s) and invoke TestSelectionController to change the selection for these. - Now both _ActionHandler OnDeselectAllTests() and OnSelectAllTests() invoke Tree.SelectAllTests(), with a parameter. - When a new TestCaseHandler is created, is registered to be notified about selection changes. It uses this notification to update the graphical node with the check. - RideTestSelectedForRunningChanged now also holds the controller of the test which has changed selection and whether has been selected or unselected. * Replace Tree DeselectAllTests(node) with SelectAllTests(node,False) as it's much more efficient. * TestSelectionController: Keep a copy of the tests to propagate in the event, rather than rebuild it everytime. * When selecting multiple tests, from the test search, do it efficiently. TestSelectionController: add select_all() to select multiple tests efficiently. Rewrite unselect_all() in terms of select_all() Tree: when selecting multiple tests, no need to navigate the full tree, just use TestSelectionController. * When deselecting multiple tests, from the test search, do it efficiently. Tree: when deselecting multiple tests, no need to navigate the full tree, just use TestSelectionController. * Do not expand all the tree nodes when selecting failed/passed tests. - Keep the information of failed/passed tests in TestCaseController - Update the information on the test execution - When selecting only failed/passed tests navigate the controller'tree rather than the graphical tree. - removed method _for_all_tests() as it was causing all the nodes to be loaded/rendered. * When selecting/deselecting multiple Tests, notify a single selection change event. Rather than one for each Test. * Restore previous behavior with "Select Failed/Passed": select only those and de-select any other test. * keep track of the selected tests using the TestCaseController(s), instead of the list of names. This removes the need for listen of name changes. * No need to generate the list of names, when determining if there's any. * Fix TestSelectionController.add_tag() * Handle RideTestSelectedForRunningChanged event on a new wx thread stack. This is to avoid the "Recursionerror: maximum recursion depth exceeded" * Remove unused variable. * Propagate the RideTestSelectedForRunningChanged event only if the selection actually changed. Co-authored-by: Valerio Bruno <v.bruno@iontrading.com> * Fix tree node (#2214) * Increase development version. * Fixes error message on RIDE Log about missing tree_node_selected. * Fixes error message on RIDE Log about missing clear_all. (#2215) * Stops log when memory exceeds limit. Co-authored-by: Helio Guilherme <helio.guilherme@load.digital> Co-authored-by: Nyral <r4nd0m4u@gmail.com> Co-authored-by: Nyral <39315668+Nyral@users.noreply.github.com> Co-authored-by: Johnny.H <jnhyperion@gmail.com> Co-authored-by: Johnny Huang <jnhuang@telenavsoftware.com> Co-authored-by: Valerio <valeriobruno@users.noreply.github.com> Co-authored-by: Valerio Bruno <v.bruno@iontrading.com>
diff --git a/requirements-dev.txt b/requirements-dev.txt
@@ -3,6 +3,7 @@ nose
 mockito
 invoke
 rellu
+memory-profiler
 # Include also normal project requirements.
 -r requirements.txt
 
diff --git a/requirements.txt b/requirements.txt
@@ -5,3 +5,4 @@ Pygments # This enables syntax highlighted in Text Editor
 robotframeworklexer # Updates lexer to newer RF 3.1.1
 robotframework
 Pypubsub
+psutil
diff --git a/src/robotide/contrib/testrunner/testrunnerplugin.py b/src/robotide/contrib/testrunner/testrunnerplugin.py
@@ -48,6 +48,7 @@
 import datetime
 import time
 import os
+import psutil
 import re
 import wx
 import wx.stc
@@ -138,6 +139,10 @@ def __init__(self, application=None):
         self._register_shortcuts()
         self._min_log_level_number = LOG_LEVELS['INFO']
         self._selected_tests: {TestCaseController} = set()
+        self._process = psutil.Process()
+        self._initmemory = None
+        self._limitmemory = None  # This will be +80%
+        self._maxmemmsg = None
 
     @property
     def _names_to_run(self):
@@ -163,6 +168,7 @@ def _copy_from_out(self, event):
             return
         if self.message_log.GetSTCFocus():
             self.message_log.Copy()
+            return
 
     def enable(self):
         self.tree.set_checkboxes_for_tests()
@@ -217,14 +223,17 @@ def _subscribe_to_events(self):
         self.subscribe(self.OnSettingsChanged, RideSettingsChanged)
 
     def OnSettingsChanged(self, data):
-        """Updates settings"""
+        '''Updates settings'''
         section, setting = data.keys
-        if section == "Test Run":  # DEBUG temporarily we have two sections
+        # print("DEBUG: enter OnSettingsChanged section %s" % (section))
+        if section == 'Test Run':  # DEBUG temporarily we have two sections
+            # print("DEBUG: setting.get('confirm run')= %s " % setting)
+            # print("DEBUG: new data= %s old %s new %s" % (data.keys, data.old, data.new))
             self.defaults.setdefault(setting, data.new)
             self.save_setting(setting, data.new)
 
     def OnTestSelectedForRunningChanged(self, message):
-        self._selected_tests = message.tests
+        self._names_to_run = message.tests
 
     def disable(self):
         self._remove_from_notebook()
@@ -267,29 +276,44 @@ def _hide_message_log(self):
         self.message_log = None
         self._right_panel.GetSizer().Layout()
 
+    def _recreate_message_log(self):
+        self.show_log_messages_checkbox.SetValue(False)
+        self._reset_memory_calc()
+        self.show_log_messages_checkbox.SetValue(True)
+
+    def _reset_memory_calc(self):
+        self._initmemory = self._process.memory_info()[0]
+        self._limitmemory = self._initmemory * 1.80
+        self._maxmemmsg = None
+
     def OnStop(self, event):
         """Called when the user clicks the "Stop" button
 
         This sends a SIGINT to the running process, with the
         same effect as typing control-c when running from the
         command line."""
+        self._reset_memory_calc()
         self._AppendText(self.out, '[ SENDING STOP SIGNAL ]\n',
                          source='stderr')
         self._test_runner.send_stop_signal()
 
     def OnPause(self, event):
+        self._reset_memory_calc()
         self._AppendText(self.out, '[ SENDING PAUSE SIGNAL ]\n')
         self._test_runner.send_pause_signal()
 
     def OnContinue(self, event):
+        self._reset_memory_calc()
         self._AppendText(self.out, '[ SENDING CONTINUE SIGNAL ]\n')
         self._test_runner.send_continue_signal()
 
     def OnStepNext(self, event):
+        self._reset_memory_calc()
         self._AppendText(self.out, '[ SENDING STEP NEXT SIGNAL ]\n')
         self._test_runner.send_step_next_signal()
 
     def OnStepOver(self, event):
+        self._reset_memory_calc()
         self._AppendText(self.out, '[ SENDING STEP OVER SIGNAL ]\n')
         self._test_runner.send_step_over_signal()
 
@@ -314,6 +338,7 @@ def OnRun(self, event):
                 if not self.ask_user_to_run_anyway():
                     # In Linux NO runs dialog 4 times
                     return
+        self._reset_memory_calc()
         self._initialize_ui_for_running()
         command = self._create_command()
         # DEBUG on Py3 it not shows correct if tags with latin chars
@@ -324,6 +349,7 @@ def OnRun(self, event):
             self._set_running()
             self._progress_bar.Start()
         except Exception as e:
+            # self._output("DEBUG: Except block test_runner.run_command\n")
             self._set_stopped()
             error, log_message = self.get_current_profile().format_error(str(e), None)
             self._output(error)
@@ -475,10 +501,19 @@ def OnTimer(self, evt):
                 self._output("\n", source="stdout")
             self._output(err_buffer, source="stderr")
         if self.message_log and not self._messages_log_texts.empty():
-            texts = []
-            while not self._messages_log_texts.empty():
-                texts += [self._messages_log_texts.get()]
-            self._AppendText(self.message_log, '\n'+'\n'.join(texts))
+            if self._process.memory_info()[0] <= self._limitmemory:
+                texts = []
+                while not self._messages_log_texts.empty():
+                    texts += [self._messages_log_texts.get()]
+                self._AppendTextMessageLog(self.message_log, '\n'+'\n'.join(texts))
+            else:
+                if not self._maxmemmsg:
+                    self._maxmemmsg = "Messages log exceeded 80% of process " \
+                                      "memory, stopping for now..."
+                    self._AppendTextMessageLog(self.message_log,
+                                               '\n' + self._maxmemmsg,
+                                               source="stderr")
+                    # self._recreate_message_log()  # DEBUG
 
     def GetLastOutputChar(self):
         """Return the last character in the output window"""
@@ -496,6 +531,7 @@ def _format_command(self, argv):
         """
         result = []
         for arg in argv:
+            # arg = arg.encode(encoding['SYSTEM'])
             if "'" in arg or " " in arg or "&" in arg:
                 # for windows, if there are spaces we need to use
                 # double quotes. Single quotes cause problems
@@ -513,6 +549,9 @@ def _show_notebook_tab(self):
             self._reload_model()
         self.show_tab(self.panel)
 
+    # from memory_profiler import profile as mprofile
+
+    # @mprofile
     def _AppendText(self, textctrl, string, source="stdout", enc=True):
         if not self.panel or not textctrl:
             return
@@ -551,6 +590,38 @@ def _AppendText(self, textctrl, string, source="stdout", enc=True):
             linecount = textctrl.GetLineCount()
             textctrl.ScrollToLine(linecount)
 
+    def _AppendTextMessageLog(self, textctrl, string, source="stdout", enc=True):
+        if not self.panel or not textctrl:
+            return
+        textctrl.update_scroll_width(string)
+        # we need this information to decide whether to autoscroll or not
+        new_text_start = textctrl.GetLength()
+        linecount = textctrl.GetLineCount()
+        last_visible_line = textctrl.GetFirstVisibleLine() + \
+                          textctrl.LinesOnScreen() - 1
+
+        textctrl.SetReadOnly(False)
+        try:
+            if enc:
+                textctrl.AppendText(string.encode(encoding['OUTPUT']))
+            else:
+                textctrl.AppendText(string)
+        except UnicodeDecodeError as e:
+            textctrl.AppendTextRaw(string.encode(encoding['SYSTEM']))
+        except UnicodeEncodeError as e:
+            textctrl.AppendText(string.encode('utf-8'))  # DEBUG .encode('utf-8'))
+
+        new_text_end = textctrl.GetLength()
+
+        textctrl.StartStyling(new_text_start, 0x1f)
+        if source == "stderr":
+            textctrl.SetStyling(new_text_end-new_text_start, STYLE_STDERR)
+
+        textctrl.SetReadOnly(True)
+        if last_visible_line >= linecount-4:
+            linecount = textctrl.GetLineCount()
+            textctrl.ScrollToLine(linecount)
+
     def _get_console_width(self):
         # robot wants to know a fixed size for output, so calculate the
         # width of the window based on average width of a character. A
