Add new statistics by karinaA7 · Pull Request #683 · Desbordante/desbordante-core

karinaA7 · 2026-02-25T13:37:16Z

New statistics:

whitespaceOnlyCount - the number of lines that consist only of whitespace characters
(space and tab)

firstCharFrequency / lastCharFrequency - the most common first/ last character in the column lines and the number of its occurrences

leadingWhitespaceCount / trailingWhitespaceCount - the number of lines with spaces at the beginning or at the end

specialCharsCount
Number of lines containing special characters

xJoskiy · 2026-02-26T19:40:06Z

Since you've added new statistic methods to DataStats, they are now printed whenever data_stats.get_all_statistics_as_string() is called in data_stats.py. So in order to pass CI you may copy output of data_stats.py into corresponding snapshot in snap_test_examples_pytest.py

xJoskiy · 2026-02-27T21:22:39Z

src/core/algorithms/statistics/data_stats.cpp

+
+        for (char c : str) {
+            if (special_chars.find(c) != std::string::npos) {
+                has_special = true;
+                break;
+            }
+        }
+
+        if (has_special) {
+            count++;
+        }


At first

Suggested change

for (char c : str) {

if (special_chars.find(c) != std::string::npos) {

has_special = true;

break;

}

}

if (has_special) {

count++;

}

for (char c : str) {

if (special_chars.find(c) != std::string::npos) {

count++;

break;

}

}

Also, finding spec character in a string take O(N * M) time, N - string length, M - spec character length. You may store a 256 sized array (number of possible values of char) for checking if the current symbol of string is in this array.

Suggested change

for (char c : str) {

if (special_chars.find(c) != std::string::npos) {

has_special = true;

break;

}

}

if (has_special) {

count++;

}

static constexpr const std::array<bool, 256> map = {0};

for (char c : special_characters) {

map[static_cast<unisgned char>(c)] = true;

}

for (char c : str) {

if (map[static_cast<unisgned char>(c)]) {

count++;

break;

}

}

Unicode symbols won't affect the frequency of special characters since they start after 0x7F.

xJoskiy · 2026-02-27T21:38:54Z

src/core/algorithms/statistics/data_stats.cpp

+
+    mo::TypedColumnData const& col = col_data_[index];
+    if (col.GetTypeId() != +mo::TypeId::kString) return {};
+    std::string const special_chars = "@#$%^&!?*_+=~'-\"";


Suggested change

std::string const special_chars = "@#$%^&!?*_+=~'-\"";

static const auto constexpr special_chars = "@#$%^&!?*_+=~'-\"";

xJoskiy · 2026-02-27T21:50:37Z

src/core/algorithms/statistics/data_stats.cpp

+    char most_frequent = '\0';
+    size_t max_count = 0;
+
+    for (auto const& pair : freq_map) {


Suggested change

for (auto const& pair : freq_map) {

for (auto [c, freq] : freq_map) {

xJoskiy · 2026-02-27T22:04:44Z

src/core/algorithms/statistics/data_stats.cpp

+    return Statistic(res, &int_type, false);
+}
+
+Statistic DataStats::GetFirstCharFrequency(size_t index) const {


GetFirstCharFrequency and GetLastCharFrequency are identical, better put a common part in a separate function

github-actions

clang-tidy made some suggestions

github-actions · 2026-02-28T10:27:19Z

src/core/algorithms/statistics/data_stats.cpp

+
+    mo::TypedColumnData const& col = col_data_[index];
+    if (col.GetTypeId() != +mo::TypeId::kString) return {};
+    static constexpr std::string_view const special_chars = "@#$%^&!?*_+=~'-\"";


warning: invalid case style for static constant 'special_chars' [readability-identifier-naming]

Suggested change

static constexpr std::string_view const special_chars = "@#$%^&!?*_+=~'-\"";

static constexpr std::string_view const kSpecialChars = "@#$%^&!?*_+=~'-\"";

src/core/algorithms/statistics/data_stats.cpp:959:

- for (char c : special_chars) { + for (char c : kSpecialChars) {

github-actions · 2026-02-28T10:27:19Z

src/core/algorithms/statistics/data_stats.cpp

+        if (col.IsNullOrEmpty(i)) continue;
+
+        auto const& str = mo::Type::GetValue<std::string>(col.GetValue(i));
+        bool has_special = false;


warning: unused variable 'has_special' [clang-diagnostic-unused-variable]

bool has_special = false; ^

github-actions

clang-tidy made some suggestions

github-actions · 2026-02-28T11:22:32Z

src/core/algorithms/statistics/data_stats.cpp

+
+    mo::TypedColumnData const& col = col_data_[index];
+    if (col.GetTypeId() != +mo::TypeId::kString) return {};
+    static constexpr std::string_view const special_chars = "@#$%^&!?*_+=~'-\"";


warning: invalid case style for static constant 'special_chars' [readability-identifier-naming]

Suggested change

static constexpr std::string_view const special_chars = "@#$%^&!?*_+=~'-\"";

static constexpr std::string_view const kSpecialChars = "@#$%^&!?*_+=~'-\"";

src/core/algorithms/statistics/data_stats.cpp:958:

- for (char c : special_chars) { + for (char c : kSpecialChars) {

xJoskiy · 2026-02-28T11:46:15Z

src/core/algorithms/statistics/data_stats.cpp

+        static std::array<bool, 256> map = {0};
+        for (char c : special_chars) {
+            map[static_cast<unsigned char>(c)] = true;
+        }


Should better put it outside of loop, do not initialize on each iteration. Initialization also can be made constexpr via lambda function

Suggested change

static std::array<bool, 256> map = {0};

for (char c : special_chars) {

map[static_cast<unsigned char>(c)] = true;

}

static contexpr std::array<bool, 256> map = [&special_chars]() constexpr {

std::array<bool, 256> map = {0};

for (char c : special_chars) {

map[static_cast<unsigned char>(c)] = true;

}

return map;

}();

xJoskiy · 2026-02-28T12:12:50Z

src/core/algorithms/statistics/data_stats.cpp

+    for (auto const& [c, freq] : freq_map) {
+        if (freq > max_count) {
+            max_count = freq;
+            most_frequent = c;
+        }
+    }


Suggested change

for (auto const& [c, freq] : freq_map) {

if (freq > max_count) {

max_count = freq;

most_frequent = c;

}

}

auto const& [c, max_freq] = *std::max_element(map.begin(), map.end(), [](const auto& lhs, const auto& rhs){ return lhs.second < rhs.second; });

Also consider empty container in order to not dereference end() iterator

Also, we might want to find the least lexicographical character if frequencies are equal. It can be made with std::tie

Suggested change

for (auto const& [c, freq] : freq_map) {

if (freq > max_count) {

max_count = freq;

most_frequent = c;

}

}

auto const& [c, max_freq] = *std::max_element(map.begin(), map.end(), [](const auto& lhs, const auto& rhs) {

// frequencies in tuples are compared first, then lexicographic order of characters

return std::tie(lhs.second, lhs.first) < std::tie(rhs.second, rhs.first);

};

xJoskiy · 2026-02-28T12:50:53Z

src/core/algorithms/statistics/data_stats.cpp

+        for (char c : str) {
+            if (map[static_cast<unsigned char>(c)]) {
+                count++;
+                break;
+            }
+        }


std::any_of

xJoskiy · 2026-02-28T12:51:23Z

src/core/algorithms/statistics/data_stats.cpp

+        for (char c : str) {
+            if (c != ' ' && c != '\t') {
+                only_space_or_tab = false;
+                break;
+            }
+        }


Why didn't you use std::isspace?

xJoskiy · 2026-02-28T18:24:09Z

src/core/algorithms/statistics/data_stats.cpp

+        if (col.IsNullOrEmpty(i)) continue;
+
+        auto const& str = mo::Type::GetValue<std::string>(col.GetValue(i));
+        static std::array<bool, 256> map = []() constexpr {


constexpr, also put outside of loop

xJoskiy · 2026-03-02T21:23:32Z

src/core/algorithms/statistics/data_stats.cpp

+        for (char c : str) {
+            if (!std::isspace(static_cast<unsigned char>(c))) {
+                only_space_or_tab = false;
+                break;
+            }
+        }


Suggested change

for (char c : str) {

if (!std::isspace(static_cast<unsigned char>(c))) {

only_space_or_tab = false;

break;

}

}

if (str.empty() or std::none_of(...))

count++;

The test doesn't pass with non_of, I suggest leaving it as is

xJoskiy · 2026-03-02T21:26:10Z

src/core/algorithms/statistics/data_stats.cpp

+Statistic DataStats::GetLeadingWhitespaceCount(size_t index) const {
+    if (all_stats_[index].leading_whitespace_count.HasValue())
+        return all_stats_[index].leading_whitespace_count;
+
+    mo::TypedColumnData const& col = col_data_[index];
+    if (col.GetTypeId() != +mo::TypeId::kString) return {};
+
+    size_t count = 0;
+
+    for (size_t i = 0; i < col.GetNumRows(); i++) {
+        if (col.IsNullOrEmpty(i)) continue;
+
+        auto const& str = mo::Type::GetValue<std::string>(col.GetValue(i));
+        if (!str.empty() && std::isspace(static_cast<unsigned char>(str[0]))) {
+            count++;
+        }
+    }
+
+    mo::IntType int_type;
+    std::byte const* res = int_type.MakeValue(count);
+    return Statistic(res, &int_type, false);
+}
+
+Statistic DataStats::GetTrailingWhitespaceCount(size_t index) const {
+    if (all_stats_[index].trailing_whitespace_count.HasValue())
+        return all_stats_[index].trailing_whitespace_count;
+
+    mo::TypedColumnData const& col = col_data_[index];
+    if (col.GetTypeId() != +mo::TypeId::kString) return {};
+
+    size_t count = 0;
+
+    for (size_t i = 0; i < col.GetNumRows(); i++) {
+        if (col.IsNullOrEmpty(i)) continue;
+
+        auto const& str = mo::Type::GetValue<std::string>(col.GetValue(i));
+        if (!str.empty() && std::isspace(static_cast<unsigned char>(str.back()))) {
+            count++;
+        }
+    }
+
+    mo::IntType int_type;
+    std::byte const* res = int_type.MakeValue(count);
+    return Statistic(res, &int_type, false);
+}


Almost identical

xJoskiy · 2026-03-02T21:26:45Z

src/core/algorithms/statistics/data_stats.cpp

+    static constexpr std::string_view const kSpecialChars = "@#$%^&!?*_+=~'-\"";
+    size_t count = 0;
+
+    std::array<bool, 256> map = []() constexpr {


Suggested change

std::array<bool, 256> map = []() constexpr {

static constexpr std::array<bool, 256> map = []() constexpr {

src/core/algorithms/statistics/data_stats.cpp

github-actions

clang-tidy made some suggestions

github-actions · 2026-03-03T16:13:08Z

src/core/algorithms/statistics/data_stats.cpp

+    static constexpr std::string_view const kSpecialChars = "@#$%^&!?*_+=~'-\"";
+    size_t count = 0;
+
+    static constexpr std::array<bool, 256> map = []() constexpr {


warning: invalid case style for static constant 'map' [readability-identifier-naming]

Suggested change

static constexpr std::array<bool, 256> map = []() constexpr {

static constexpr std::array<bool, 256> kMap = []() constexpr {

src/core/algorithms/statistics/data_stats.cpp:960:

- [](char c) { return map[static_cast<unsigned char>(c)]; })) { + [](char c) { return kMap[static_cast<unsigned char>(c)]; })) {

github-actions · 2026-03-03T16:13:09Z

src/core/algorithms/statistics/data_stats.h

+    // Returns the most frequent last character.
+    Statistic GetLastCharFrequency(size_t index) const;
+    enum class CharPosition { kFirst, kLast };
+    enum class WhitespacePosition { Leading, Trailing };


warning: invalid case style for enum constant 'Leading' [readability-identifier-naming]

Suggested change

enum class WhitespacePosition { Leading, Trailing };

enum class WhitespacePosition { kLeading, Trailing };

github-actions · 2026-03-03T16:13:09Z

src/core/algorithms/statistics/data_stats.h

+    // Returns the most frequent last character.
+    Statistic GetLastCharFrequency(size_t index) const;
+    enum class CharPosition { kFirst, kLast };
+    enum class WhitespacePosition { Leading, Trailing };


warning: invalid case style for enum constant 'Trailing' [readability-identifier-naming]

Suggested change

enum class WhitespacePosition { Leading, Trailing };

enum class WhitespacePosition { Leading, kTrailing };

github-actions

clang-tidy made some suggestions

github-actions · 2026-03-03T16:34:54Z

src/core/algorithms/statistics/data_stats.cpp

+    static constexpr std::string_view const kSpecialChars = "@#$%^&!?*_+=~'-\"";
+    size_t count = 0;
+
+    static constexpr std::array<bool, 256> kmap = []() constexpr {


warning: invalid case style for static constant 'kmap' [readability-identifier-naming]

Suggested change

static constexpr std::array<bool, 256> kmap = []() constexpr {

static constexpr std::array<bool, 256> kMap = []() constexpr {

src/core/algorithms/statistics/data_stats.cpp:960:

- [](char c) { return kmap[static_cast<unsigned char>(c)]; })) { + [](char c) { return kMap[static_cast<unsigned char>(c)]; })) {

xJoskiy · 2026-03-04T17:14:26Z

src/core/algorithms/statistics/data_stats.h

+    enum class CharPosition { kFirst, kLast };
+    enum class WhitespacePosition { kLeading, kTrailing };


Seems like the are serving the same purpose. You might leave only the CharPosition enum and place it somewhere above, closer to DataStats() ctor, not in the middle of methods declaration

xJoskiy · 2026-03-07T12:54:12Z

src/core/algorithms/statistics/data_stats.cpp

+    return GetWhitespaceCount(index, CharPosition::kLast);
+}
+
+Statistic DataStats::GetSpecialCharsCount(size_t index) const {


The naming does not reflect the actual purpose of this function. Should better name it like GetNumberOfRowsWithSpecialChars

xJoskiy · 2026-03-07T12:55:53Z

src/core/algorithms/statistics/data_stats.cpp

+Statistic DataStats::GetLeadingWhitespaceCount(size_t index) const {
+    return GetWhitespaceCount(index, CharPosition::kFirst);
+}
+
+Statistic DataStats::GetTrailingWhitespaceCount(size_t index) const {


The same with these functions. GetNumberOfRowsWithTrailingWhitespaces

karinaA7 force-pushed the add-new-statistics branch from c5c3d3d to a978811 Compare February 25, 2026 13:47

karinaA7 force-pushed the add-new-statistics branch from a978811 to df3b75f Compare February 26, 2026 20:18

xJoskiy reviewed Feb 27, 2026

View reviewed changes

karinaA7 force-pushed the add-new-statistics branch 2 times, most recently from 9dab0de to 742d809 Compare February 28, 2026 10:21

github-actions bot reviewed Feb 28, 2026

View reviewed changes

karinaA7 force-pushed the add-new-statistics branch from 742d809 to 8ef7d19 Compare February 28, 2026 11:16

github-actions bot reviewed Feb 28, 2026

View reviewed changes

xJoskiy reviewed Feb 28, 2026

View reviewed changes

karinaA7 force-pushed the add-new-statistics branch from 8ef7d19 to c5b7a1b Compare February 28, 2026 12:11

xJoskiy reviewed Feb 28, 2026

View reviewed changes

xJoskiy self-assigned this Feb 28, 2026

xJoskiy reviewed Feb 28, 2026

View reviewed changes

karinaA7 force-pushed the add-new-statistics branch from c5b7a1b to 03a3e42 Compare March 2, 2026 15:36

xJoskiy reviewed Mar 2, 2026

View reviewed changes

src/core/algorithms/statistics/data_stats.cpp Show resolved Hide resolved

karinaA7 force-pushed the add-new-statistics branch from 03a3e42 to b52df00 Compare March 3, 2026 16:07

github-actions bot reviewed Mar 3, 2026

View reviewed changes

karinaA7 force-pushed the add-new-statistics branch from b52df00 to fe1f418 Compare March 3, 2026 16:27

github-actions bot reviewed Mar 3, 2026

View reviewed changes

karinaA7 force-pushed the add-new-statistics branch 2 times, most recently from 9237253 to d848454 Compare March 3, 2026 20:33

xJoskiy reviewed Mar 4, 2026

View reviewed changes

Add new statistics

0e8419f

karinaA7 force-pushed the add-new-statistics branch from d848454 to 0e8419f Compare March 5, 2026 11:07

xJoskiy reviewed Mar 7, 2026

View reviewed changes

	std::string const special_chars = "@#$%^&!?*_+=~'-\"";
	static const auto constexpr special_chars = "@#$%^&!?*_+=~'-\"";

	for (auto const& pair : freq_map) {
	for (auto [c, freq] : freq_map) {

	static constexpr std::string_view const special_chars = "@#$%^&!?*_+=~'-\"";
	static constexpr std::string_view const kSpecialChars = "@#$%^&!?*_+=~'-\"";

	std::array<bool, 256> map = []() constexpr {
	static constexpr std::array<bool, 256> map = []() constexpr {

	static constexpr std::array<bool, 256> map = []() constexpr {
	static constexpr std::array<bool, 256> kMap = []() constexpr {

	enum class WhitespacePosition { Leading, Trailing };
	enum class WhitespacePosition { kLeading, Trailing };

		enum class CharPosition { kFirst, kLast };
		enum class WhitespacePosition { kLeading, kTrailing };

Conversation

karinaA7 commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xJoskiy commented Feb 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xJoskiy Feb 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Feb 28, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Feb 28, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Feb 28, 2026

Choose a reason for hiding this comment

Uh oh!

xJoskiy Feb 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xJoskiy Feb 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xJoskiy Feb 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

karinaA7 commented Feb 25, 2026 •

edited

Loading

xJoskiy Feb 28, 2026 •

edited

Loading

xJoskiy Feb 28, 2026 •

edited

Loading

xJoskiy Feb 28, 2026 •

edited

Loading

xJoskiy Feb 28, 2026 •

edited

Loading

xJoskiy Mar 7, 2026 •

edited

Loading