All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Please mark backwards incompatible changes with an exclamation mark at the start.
- A
NameErrorthat was being raised whenjay_api/elasticsearch/clientwas required without requiringelasticsearch. - A
NoMethodErrorthat was being raised byElasticsearch::Stats::Indiceswhenactive_support/core_ext/stringhadn't been loaded.
- The
#force_mergemethod to theElasticsearch::Indexclass. This method starts a Forced Segment Merge on the index. - The
#totalsmethod toElasticsearch::Stats::Index, this gives the caller access to the index's total metrics. - The
Elasticsearch::Stats::Index::Totalsclass. The class contains information about an index's total metrics, for example, total number of documents, total size, etc. - The
#settingsmethod to theElasticsearch::Indexclass. This gives the caller access to the index's settings. - The
Elasticsearch::Indices::Settings::Blocksclass. The class encapsulates an index's blocks settings (for example, whether the index is read-only). - The
Elasticsearch::Indices::Settingsclass. The class encapsulates an index's settings. - It is now possible to configure the type used by the
RSpec::TestDataCollectorclass when pushing documents to Elasticsearch. If no type is specified in the configuration the default type will be used. - Allow the
Elasticsearch::IndexandElasticsearch::Indexes's#pushmethod to receive atypeparameter, just like#indexdoes.
- Support for the
bucket_selectorpipeline aggregation inElasticsearch::QueryBuilder::Aggregations. This allows filtering buckets based on computed metrics (e.g., filtering terms buckets by aggregated values).
PropertiesFetcher#lastnow correctly returns the last set of properties (ordered chronologically).
- The
timeoutparameter toElasticsearch::ClientFactory#create. The parameter allows the user to specify the timeout in seconds for Elasticsearch requests.
- A
#clonemethod toElasticsearch::QueryBuilderthat properly clones theQueryBuilderand its nested objects. - ActiveSupport's
#present?,#presenceand#blank?methods can now be used in ERB configuration files.
- The
#boolmethod to theQueryBuilder::QueryClauses::Boolclass. This allows boolean clauses to be nested. QueryBuilder#sortcan now receive either the direction of the sorting (ascordesc) or aHashwith advanced sorting options. These are relayed directly to Elasticsearch.
- ! Updated the
gitdependency from~> 1, >= 1.8.0-1to~> 3. - ! Increased the minimum Ruby version requirement to 3.1.0
- The
Elasticsearch::Indexesclass. A class which allows multiple indexes to be used (fed or queried) at the same time.
- The
Aggregations::Compositeclass and theAggregations#compositemethod. They make it possible to use Elasticsearch'scompositeaggregations.
- The
#nodesmethod to theElasticsearch::Statsclass. This method gives the user access to the node-related statistics of the Elasticsearch cluster.
- The
#statsmethod to theElasticsearch::Clientclass. The method returns an object that can be used to retrieve statistics about the Cluster. For the moment only#indicesis available, which returns index-related statistics.
-
! The
#keysmethod to theConfigurationclass. The method returns the array of keys that theConfigurationobject has.Note that the addition of this method means that it is no longer possible to access the value of an attribute called
keysvia the dot syntax, however, it is still possible to access its value using the brackets:[]
- The
Aggregations::TopHitsclass cloning. The nested aggregations are now also being cloned as expected.
- The
Aggregations::DateHistogramclass and theAggregations#date_histogrammethod. They make it possible to use Elasticsearch'sdate_histogramaggregations.
- The
#pingmethod toElasticsearch::Client
- The
Aggregations::Cardinalityclass and theAggregations#cardinalitymethod. They make it possible to use Elasticsearch'scardinalityaggregations.
GitilesHelper#gitiles_urlcan now be called without apath. When no path is given the method generates a link to the givenrefspecinstead.
PropertiesFetcher#lastnow correctly returns the last set of properties (ordered chronologically).
QueryBuilder#sourcecan now takefalse,ArrayandHashin addition to simple strings.
- The
maxmethod to theQueryBuilders::Aggregationsclass. - Added the
Aggregations::Maxclass. To model Elasticsearch'smaxaggregation type.
- !
QueryClauses#mergeno longer produces nested boolean clauses when two boolean queries are merged.
- The
#negateand#negate!methods to theQueryClausesclass. - The
QueryClauses::Negatorclass. - The
QueryClauses::MatchNoneclass and the corresponding#match_nonemethod to theMatchClausesmodule. This allows the use of Elasticsearch's match_none query to be used with the Query Builder - The
QueryClauses::MatchAllclass and the corresponding#match_allmethod to theMatchClausesmodule. This allows the use of Elasticsearch's match_all query to be used with the Query Builder #mergeand#merge!to theQueryClauses:Boolclass
Elasticsearch::Index#delete_by_query_async, a method that asynchronously deletes the documents matching a query from the indexJayAPI::Elasticsearch::Async, a class that provides functionality to perform asynchronous operations on an Elasticsearch indexJayAPI::Elasticsearch::Tasks, a class that represents Elasticsearch tasks- Two additional optional parameters:
slicesandwait_for_completiontoElasticsearch::Index#delete_by_query. QueryBuilder::QueryClauses::MatchClauses#regexp, a method that adds aJayAPI::Elasticsearch::QueryBuilder::QueryClauses::Regexpclause to the Query Clauses set.QueryBuilder::QueryClauses::Regexp, a class that represents a Regexp query in Elasticsearch.
- The
typeparameter toElasticsearch::Index's#indexmethod. The parameter can be set tonilto avoid the creation of nested documents. - The
Aggregations::TopHitsclass and theAggregations#top_hitsmethod. They make it possible to use Elasticsearch'stop_hitsaggregations.
- The
clonemethod to theQueryBuilder::Aggregationsclass.
- Merging two
QueryBuilderinstances now preserves nested aggregations. QueryBuilder::Aggregations::ValueCountno longer accepts nested aggregations.QueryBuilder::Aggregations::Sumno longer accepts nested aggregations.
QueryBuilder::Aggregations::Avgno longer accepts nested aggregations.Elasticsearch::Clientwill no longer retry requests when they fail with one of the following errors:BadRequest,Unauthorized,Forbidden,Not Found,MethodNotAllowed,RequestEntityTooLarge,NotImplemented.
- The
Aggregations::ValueCountclass and theAggregations#value_countmethod. They make it possible to use Elasticsearch'svalue_countaggregations.
- The
Aggregations::Filterclass and theAggregations#filtermethod. They make it possible to use Elasticsearch'sfilteraggregations.
- ! The return value of
Aggregations#to_h. Instead of returning only the aggregations themselves as aHashthe method now returns aHashwith a rootaggskey under which the actual aggregations are placed.
- The
Aggregations::Sumclass andAggregations#summethod. They allow Elasticsearch'ssumaggregation to be used. - The
clonemethod to theAggregationclasses. - The
aggsmethod to theAggregationclasses. This allows Elasticsearch aggregations to be nested allowing the creation of composite aggregations. - The
none?method to theAggregationsclass.
! The CLI. The Gem no longer offers a CLI. The CLI functionality has been
moved the jay_cli gem. Please install that gem instead.
- An issue in the
PropertiesFetcherclass'slastandfirstmethods which produced incorrect results when thebuild_numberfield resets, for example, when build jobs are moved or re-created.
- ! The following methods from the
QueryBuilder::Aggregationsclass:add,clear,[]
- The
terms,avgandscripted_metricmethods to theQueryBuilders::Aggregationsclass. These replace the formeraddmethod. - Added the
Aggregations::Avgclass. To model Elasticsearch'savgaggregation type. - Added the
Aggregations::ScriptedMetricclass. To model Elasticsearch'sscripted_metricaggregation type. - Added the
Aggregations::Termsclass. To model Elasticsearch'stermsaggregation type. - Added the
QueryBuilder::Scriptclass. The class represents ascripted element in an Elasticsearch query (can be used with some query clauses and aggregations).
- Added the
mergemethod toQueryBuilder - Added the
mergemethod toAggregations - Added the
mergemethod toQueryClauses - Added the
clonemethod toQueryClauses - Added the
empty?method toQueryClauses - Added the
clonemethod toQueryClauses::Bool
- ! Removed the
query=method fromQueryClauses::QueryString - ! Removed the
conditionsmethod from theQueryBuilderclass.
- Added a missing
requireoffaraday/errortoJayAPI::Elasticsearch::Client
- ! The
PriorVersionFetcherBaseclass no longer accepts acluster_urland aportparameter, it now requires an already-initializedJayAPI::Elasticsearch::Clientobject.
- Added the
delete_by_querymethod to theElasticsearch::Indexclass. - Added the
QueryClauses::Termsclass and the corresponding#termsmethod to theMatchClausesmodule. These allow Elasticsearch's Terms query to be used when making use of theQueryBuilderclass.
- Enhanced
ClientFactory#createwith new configurable parameters:max_attempts,wait_strategyandwait_time.
- A
Responseclass that wraps around the raw Elasticsearch response Hash - A
BatchCounterclass that keeps track of which batch of results the QueryResults object is for. - A
SearchAfterResultsclass that is a QueryResults class but for the Elasticsearch 'search_after' query type. - Added 'search_after' param to Index#search, as a replacement for Scroll API.
- ! Scroll API feature is removed, because it is deprecated.
- Fixed a
NoMethodErrorthat was introduced withactivesupport 7.1.0.
- Add missing double splat operators to the calls for methods with keyword
arguments. This fixes the
ArgumentErrorbeing raised when running with Ruby 3.0+ - Fixed the
Elasticsearch::ClientFactoryclass's type-hint documentation.
- ! Changed the
PropertiesFetcher#allmethod:- The method can now take a block, if a block is given each of the fetched documents will be yielded to the given block.
- If no block is given then an
Enumeratoris returned. Which can be used to iterate or access each of the individual elements of the fetched set. - The method no longer returns a
QueryResultsobject.
- Added
by_sut_revisionmethod toPropertiesFetcherclass. This allows build properties to be fetched by specifying the desired 'sut_revision'.
- Requirements in RSpec tests whose results are being collected with the
TestDataCollectorclass can now be annotated with eitherrequirementsorrefs.
- Fixed a set of
ArgumentErrors that were happening during the initialization of theJayAPI::Elasticsearch::Index.
- A wrapper class
JayAPI::Elasticsearch::Clientis introduced to wrap overElasticsearch::Clientclass. Its API is the same as the original Client, but it will now retry a request a few times if one of the following errors are raised:Elasticsearch::Transport::Transport::ServerErrorFaraday::TimeoutError
- !The Index class's constructor changed its signature to take an already initialized JayAPI::Elasticsearch::Client object, instead of the parameters needed to initialize it.
- ! Removed the
-bswitch from the J-Unit and C-Dash parsers. The parsers will no longer read build properties files.
- ! Changed the J-Unit and C-Dash parsers, they will no longer read environment variables from the environment, instead they require the Test Environment to be specified in the configuration.
- ! Updated
activesupportfrom~> 6to~> 7 - ! Updated
dredgefrom~> 0to~> 2
- ! Set the minimum requirement for Ruby to 2.7.0
- Restricted the version of the
oxgem to<= 2.14.14
- ! Changed the
JayAPI::PropertiesFetcherclass. Instead of receiving all the parameters to initialize an index internally it now takes an already initializedJayAPI::Elasticsearch::Indexobject.
- Fixed a bug in the
Elasticsearch::Indexclass which caused the error message to be empty when an error occurred during push and the error details were not found in the first element of the response body.
- Added the
jsonas subcommand to the CLI'simportcommand.
- The
TestDataCollectorclass will now push the remote URL of the repository from which the tests are running to Elasticsearch.
- Added the
TimestampHandlerclass for the Parsers. - Added the
JSON::Parserclass. - Added a
#to_yamlmethod toConfiguration, which allows to print the entire configuration in a human readable YAML format. - If a search query causes an error, the corrupt query will be logged.
- Improved the timestamp handling in the JUnit parser. The parser is now capable
of parsing JUnit files in which the
timestampattribute is given in thetestsuitetag, thetestcasetag or both.
- Added the possibility to use the
-mcommand line switch when importing JUnit tests to provide test meta-data. The meta-data is expected to be stored in JSON files following ESR Labs's custom format for test meta-data.
- Added functionality to merge, with specific rules,
JayAPI::Configurationobjects.
- Fixed a bug in the methods
by_build_jobandby_software_versionin thePropertiesFetcherclass that caused the wrong data to be fetched from Elasticsearch for certain queries.
- Adapted the JUnit and CDash parsers to make them take the Test Environment from the Configuration instead of taking it from a hardcoded set of environment variables (backwards compatible).
- Added a
deep_to_hmethod to theJayAPI::Configurationclass. - Added a
with_indifferent_accessmethod to theJayAPI::Configurationclass.
- Extended the CDash parser implementation to allow it to use the Dredge gem to extract meta-data from the source files and attach it to the test results before pushing them to Elasticsearch.
- Removed the
jira-rubyruntime dependency.
- Updated activesupport to version 6.x
- Changed the minimum Ruby version required by the gem to 2.5.0 (because of the
activesupportgem update).
- Added a public Repository#open_or_clone! method.
- Added the JayAPI::Git::Repository#remote_url method.
- Updated the
elasticsearchgem to version 7.9.0 - Changed the minimum Ruby version required by the gem to 2.4.0 (because of the
elasticsearchgem update).
- Added the
cdashsubcommand to theimportcommand that allows to parse CDash XML files and import them to Elasticsearch.
- Changed the
QueryStringclause to allow the user to omit thefieldsparameter.
- Added the
Existsquery clause to Elasticsearch'sQueryBuilder - Added the
Termquery clause to Elasticsearch'sQueryBuilder - Added the
Rangequery clause to Elasticsearch'sQueryBuilder
- Moved the functions
document,each, andinitializeto the base classParsers::Parserso that both,CDash::ParserandJUnit::Parsercan also use them. - Changed the attribute
version_arraytoexistent_versionsfor readability purposes for the classPriorVersionFetcherBase.
- Added a new parser
CDash::Parserthat is able to parse XML files in CDash format. - Added multiple classes
CDash::TestObject,CDash::Testingthat are able to parse specific CDash XML tags and extract information from them. - Added a class
CDash::TestSuitethat is responsible for collectingCDash::Testobjects and compute various information using those.
PriorVesionFetcherBaseused the functionqueryfromQueryBuilderwhich has changed its meaning. Fixed the issue by usingto_queryinstead.
- Added the
beforemethod to thePropertiesFetcherclass.
- Changed the
aftermethod of thePropertiesFetcherclass to use>instead of>=when composing the query string (mainly to keep the two methods consistent).
- Allows the
aftermethod from thePropertiesFetcherto takeStrings as arguments. The string should of course be in format expected by Elasticsearch.
- Added the
aftermethod to thePropertiesFetcherto fetch only properties that were pushed after the given timestamp.
- Added authentication params to the
Elasticsearch::Indexclass constructor. It is now possible to specify authentication credentials for the connection to the Elasticsearch Clusters. The change is backwards compatible, if the cluster has no authentication these parameters can be omitted altogether.
- Refactored the
PropertiesFetcherclass to make it more flexible. The Class now allows the combination of multiple conditions and allows the caller to decide if it wants the first, the last or all the Build Properties records.
- Added a new class
PriorVersionFetcherBasewhich is responsible for fetching the previous software version from the one provided. For that it checks all the versions on Jay to return a prior version that actually exists. - Added the
MatchPhrasequery clause.
(yanked)
- Changed the name of the file that contains the
PropertiesFetcherclass to match the name of the class. - Changed the name of the
indexparameter of thePropertiesFetcherclass's constructor toindex_nameto make it consistent with the same parameter in theElasticsearch::Indexclass. - Changed the way the
QueryBuilderfor Elasticsearch handles query clauses. These are now handled by theQueryClausesclass, which allows for multiple types of query clauses. This also makes the API exposed by theQueryBuilderclass backwards incompatible.
- Modified the J-Unit parser to attach the build properties
version_codeandversion_nameto each test suite.
- Added the
--versionswitch to the CLI.
- Expanded Jay API's J-Unit parser to make it work with the J-Unit files produced
by Gradle (this requires the build job name and number, the project name and
the version code to be provided via different means). Environment variables
and
build.proprtiesfiles are supported for this purpose.
- Removed all the
JayAPI::JIRAmodules and tests due to their migration to theJIRATamergem. - Change the way the
JayAPI::IDBuilderclass generates short IDs for test cases. Instead of removing all special characters from the string before computing the hash hyphens are now kept. This produces different Short IDs for test cases whose names differ only by a minus sign.
- Functionality to access the Elasticsearch Scroll API. This allows to surpass the allowed query limit of data (default 10k docs).
- Added the Aggregations feature to the Elasticsearch classes. This means:
- The
QueryBuilderclass can now take aggregations and add them to the composed query. - The
QueryResultsobject is now aware of the possibility that a Query result may contain anaggregationskey and will provide direct access to it. (The implementation here is very crude). - Added the
Aggregationsclass to manipulate aggregations when working with aQueryBuilder.
- The
- Added the
PropertiesFetcherclass that is responsible for fetching build properties from JAY when provided with proper configuration.
- Added the
remotesmethod to theJayAPI::Git::Repositoryclass. The method returns the list of Remote Repositories linked to the repository. - Added the
GitilesHelpermodule. A module with methods to handle Gerrit's Gitiles URLs.
- A bug in the
JIRA::CachedIssueBrowser. When checking for known JIRA issues for a specific test case id, it would check whether the string value of the issue inside the ticket includes the specified test case id with 'include?' method. This would only be the case when they are identical. For a lot of cases, the test case id specified in a JIRA ticket is shorter and more general, so that it could encompass many different test cases that share a common namespace. In those cases, however, the current implementation would discard the relavant jira tickets, because their annotated test case ids are shorter than the specified one, and hence could not include it. - Another issue in
JIRA::CachedIssueBrowser. The class would split test case ids inside a JIRA issue by using a regular expression. This would generate a list, with different test case ids. However, sometimes empty elements would show up in the list which are now filtered out.
- Added a 'checkout' method to the JayAPI::Git::Repository class.
-
Added the
JIRA::CachedIssueBrowserclass. This class is a drop-in replacement for the standardJIRA::IssueBrowserclass. The major difference is that this class fetches all the issues and builds a local cache with them, then it performs a local search on the cache to find the issues related to a given Test Case ID.This is needed when a big number of failed test cases are expected. Querying JIRA for each failed test case takes a huge amount of time whereas searching the local cache is a lot faster, albeit it requires more memory and processing power on the machine executing the search.
- Fixed a bug in the
JIRA::IssueBrowserclass that caused the class to always use the fieldTestcase_IDwithout any regard for the given field name. This means that with a different field name issues would be found but never returned because they would be discarded by thefilter_issuesmethod.
- Updates the
thorgem to Version 1.1.0. The changes betweenthor0.x and 1.x do not affectjay_apidirectly as it is already compliant. But other gems might need to add theexit_on_failure?method to their CLI classes. - Documents the possibility of passing
nilon theurlparameter of theRepositoryclass' constructor. Adjusts the class to react correctly under such circumstances. Tests were also added to verify the behaviour.
- Added the
TestDataCollectorclass. This class is a Formatter forRSpecthat allows the pushing of test results to Jay's Elasticsearch backend. Its main purpose is to push the test results and in particular the annotated requirements for projects whose tests are running on pureRSpec(i.e. not inside Elite).
- Moved
rspecfrom the development to the runtime dependencies. This is needed because RSpec is now being included directly by one of the classes exposed by the API.
- Removed the security feature for the ERB evaluation on configuration files.
This was causing a
SecurityErrorto be raised even when the statements inside the files were perfectly safe. The feature will be disabled until the root cause for the problem can be established and addressed.
- Allow configuration files to use ERB, this allows (among other things) the use of environmental variables inside the configuration files.
- Add the
configuration_fileandcheck_configrationmethods to theJayAPI::CLI::Baseclass. This methods allows the given configuration file to be checked before it is loaded. Only basic checks are performed.
- Added a
version_clausemethod to theQueryComposerclass. The method allows subclasses to add a clause to restrict the query results to a particular software version, branch, or in the case of certain OEMs a specific cluster.
JIRA::IssueBrowserwill now catchSocketErroras well.
- Allows the
IssueBrowserclass to receive aHashof options in addition to aQueryComposerclass. This hash of options is then passed down to the givenQueryComposerclass' constructor during initialization.
Elasticsearch::QueryResults#allcan now be called without a block. In this case the method will return anEnumeratorthat can be used to iterate over the whole set of documents. AnEnumeratoris also anEnumerableso the whole spectrum of collection-based Ruby methods will be available.
- Added a new module to elasticsearch that stores the time format recognized by elasticsearch and provides a function that transforms time into this format.
- Added the
sourcemethod to theJayAPI::Elasticsearch::QueryBuilderclass to allow the user to extract only a subset of the document from Elasticsearch.
- Added the
branchesmethod to theJayAPI::Git::Repositoryclass to allow the user to fetch the list of available branches. - Added the
logmethod to theJayAPI::Git::Repositoryclass to allow the user to get the list of commits in the current branch or a given branch. - Added the
add_worktreemethod to theJayAPI::Git::Repositoryclass to allow the user to create worktrees out of a repository. - Added the
worktreesmethod to theJayAPI::Git::Repositoryclass to allow the user to manipulate worktrees, list them, and remove them.
- Changed JIRA's
QueryComposerclass to filter out rejected tickets from the results.
-
Added the
objectmethod to theJayAPI::Git::Repositoryclass to allow the user to fetch commit information by providing the commitSHA1or another Git reference (tag, branch, etc). -
Added the
allmethod to theJayAPI::Elasticsearch::QueryResultsclass, the method allows the caller to easily iterate over all the records, not only the ones in the currentQueryResultsobjects but all the available records until the whole result set has been traversed.This is a very common task, therefore the code was abstracted to the API.
- Fixed a bug in the
JayAPI::Elasticsearch::QueryBuilderclass that was causing strings to be improperly escaped when building the query. The quotation marks don't need to be escaped twice because this is being handled by the JSON library when converting the queryHashto a JSON string.
- Changed the
JayAPI::Configurationclass to allow it to parse Hashes that are nested inside arrays.
- Added the necessary logic to generate the Short ID (
id) and the Secure ID (id_secure) for the Imported JUnit Test Cases
- Modifies the JUnit Parser it now receives a
project_name, it uses it to generate the Test Case ID.
- Added the
collapsemethod to theElasticsearch::QueryBuilderclass. - Added some convenient methods to the
Elasticsearch::QueryResultsclass:first,last,any?
- Moved the
empty?method on theElasticsearch::QueryResultsclass to the delegator.
- Added the
RequirementsExtractorclass for the JUnit Parser. The Class is able to parse the output of the Test Suites and extract the requirements for each of the Test Cases from it. - Added the
IDBuilderclass to generate Short and Secure IDs for the Test Cases.
- Fixed a dependency issue with the
Oxgem. The gemspec was requesting the Gem on version >= 2.13.2 which was not really required and was causing issues when trying to use thejay_apiGem inside Elise.
- Fix a bug causing a
NoMethodErrorwhen calling thenext_batchmethod in theQueryResultsclass. The error was occurring because thesearchmethod in theIndexclass is returning an instance of theQueryResultsclass instead of a simple Hash. This change is backwards incompatible because the method no longer updates the state of the current instance but returns a different instance instead.
- Added the Import/Parse feature for J-Unit files:
- CLI for importing J-Unit files.
- Parsing and importing logic.
- Locked the version of the elasticsearch gem to be less than
7.6.0(because that version requires Ruby >= 2.4.0) - Fixed a bug that caused the Repository class to clone the repositories in the wrong location (the name of the repository was being added twice to the path because of the way the git gem works).
- Made the second parameter for
JayAPI::Errors::ConfigurationErroroptional.
- Added the Git::Repository class which features methods to perform lazy cloning and fetching of git repositories and to update they as needed.
- Added Unit Tests for the
fetch_ticketmethod.
- Changed the
fetch_ticketmethod in theJayAPI::JIRA::IssueBrowserclass to raise an error when an invalid JIRA Ticket ID is given.
- Added the IssueBrowser and QueryComposer classes for JIRA, they provide functionality to search for issues by TestCase ID or to fetch an issue by its ID.
- Added the Elasticsearch related classes. They provide access to Elastic Search functions and provide an easy interface to query, search and insert records into an Elasticsearch Index