Function updates, mostly fixing get/set solver data

Author: GearsAD

docs/src/BuildingGraphs.md

@@ -62,7 +62,9 @@ DFG and IIF rely on a CRUD (Create, Read, Update, and Delete) interface to allow
 Variables are added using IncrementalInference's `addVariable!` function. To create the variable, you provide the following parameters:
 - The graph the variable is being added to
 - The variable's label (e.g. :x1 or :a)
-- The variable type (which is a subtype of InferenceVariable)
+- The variable inference type (aka soft type), which is a subtype of InferenceVariable
+
+**Note**: Once variables are initialized to a specific soft type, all variable node data (solver data) must use that type. 
 
 In addition, the following optional parameters are provided:
 - Additional labels for the variable (in DFG these are referred to as tags)

docs/src/GraphData.md

@@ -1,185 +1,167 @@
-# Creating Graphs
+# Using Graph Elements
 
-In this section constructing DFG graphs will be discussed. To start, bring DistributedFactorGraphs into your workspace:
+Variables and factors in DistributedFactorGraphs are used for a variety of
+different applications. We have tried to compartmentalize the data as much as
+possible so that users do not need to dig around to find what they need (it's a work in progress).
 
-```julia
-using DistributedFactorGraphs
-```
-
-We recommend using IncrementalInference (IIF) to populate DFG graphs. DFG provides the structure, but IIF overloads the provided `addVariable!` and `addFactor!` functions and creates solver-specific data that allows the graph to be solved. So although you can use DFG's `addVariable!` and `addFactor!`, it is better to start with IIF's functions so that the graph is solvable.
+There are three fundamental types of data in DFG:
+- Variable and factor data (stored in the nodes themselves)
+- Offloaded big data elements (keyed in a variable or factor, but stored in another location)
+- Graph data (data that is related to the graph itself)
 
-So for the following examples, IncrementalInference will be used to create the variables and factors. It should be added and imported to run the examples:
+The following is a guideline to using these parameters.
 
-```julia
-using Pkg
-Pkg.add("IncrementalInference")
-using IncrementalInference
-```
+> Note: Some functions are direct accessors to the internal parameters, others are derived functions (e.g. getLabel(v) = v.label). In other cases the accessors are simplified ways to interact with the structures. We recommend using the accessors as the internal structure may change over time.
 
-## Initializing a Graph
+> Note: Adds in general throw an error if the element already exists. Update will update the element if it exists, otherwise it will add it.
 
-DFG graphs can be built using various drivers (different representations of the underlying graph). At the moment DFG supports 3 drivers:
-- GraphsDFG: An in-memory graph that uses Graphs.jl for representing the graph.
-- LightDFG: An in-memory graph that uses LightGraphs.jl for representing the graph.
-- CloudGraphs: A database-driven graph that uses Neo4j.jl for interacting with the graph.
+> Note: In general these functions will return an error if the respective element is not found. This is to avoid returning, say, nothing, which will be horribly confusing if you tried `getVariableSolverData(dfg, :a, :b)` and it returned nothing - which was missing, :a or :b, or was there a communication issue? We recommend coding defensively and trapping errors in critical portions of your user code.
 
-In general the first two are used for building and solving graphs, and CloudGraphs is used for persisting in-memory graphs into a database. In the long term we recommend using the LightDFG driver for in-memory operation because Graphs.jl is not actively supported and over time that driver may be deprecated.
-
-To continue the example, run one of the following to create a DFG driver:
-
-### Creating a GraphsDFG Graph
-
-```julia
-# Create a DFG with default solver parameters using the Graphs.jl driver.
-dfg = GraphsDFG{SolverParams}(params=SolverParams())
-```
+> Note: All data is passed by reference, so if you update the returned structure it will update in the graph. The database driver is an exception, and once the variable or factor is updated you need to call update* to persist the changes to the graph.
 
-### Creating a LightDFG Graph
+The following examples make use this data:
 
 ```julia
+using IncrementalInference
 # Create a DFG with default solver parameters using the LightGraphs.jl driver.
 dfg = LightDFG{SolverParams}(params=SolverParams())
-```
-
-### Creating a CloudGraphsDFG Graph
 
-```julia
-# Create a DFG with no solver parameters (just to demonstrate the difference) using the CloudGraphs driver, and connect it to a local Neo4j instance.
-dfg = CloudGraphsDFG{NoSolverParams}("localhost", 7474, "neo4j", "test",
-                                "testUser", "testRobot", "testSession",
-                                nothing,
-                                nothing,
-                                IncrementalInference.decodePackedType,
-                                IncrementalInference.rebuildFactorMetadata!)
+x0 = addVariable!(dfg, :x0, ContinuousScalar, labels = [:POSE], solvable=1)
+x1 = addVariable!(dfg, :x1, ContinuousScalar, labels = [:POSE], solvable=1)
+f1 = addFactor!(dfg, [:x0; :x1], LinearConditional(Normal(50.0,2.0)), solvable=1)
 ```
 
-## Creating Variables and Factors
+## Variable and Factor Elements
 
-DFG and IIF rely on a CRUD (Create, Read, Update, and Delete) interface to allow users to create and edit graphs.
+### Common Elements
 
-### Creating Variables with IIF
+#### Labels
 
-Variables are added using IncrementalInference's `addVariable!` function. To create the variable, you provide the following parameters:
-- The graph the variable is being added to
-- The variable's label (e.g. :x1 or :a)
-- The variable type (which is a subtype of InferenceVariable)
+Labels are the principle identifier of a variable or factor.
 
-In addition, the following optional parameters are provided:
-- Additional labels for the variable (in DFG these are referred to as tags)
-- A `solvable` flag to indicate whether the variable is ready to be added to a solution
-
-Three variables are added:
+```@docs
+getLabel
+```
 
-```julia
-v1 = addVariable!(dfg, :x0, ContinuousScalar, labels = [:POSE], solvable=1)
-v2 = addVariable!(dfg, :x1, ContinuousScalar, labels = [:POSE], solvable=1)
-v3 = addVariable!(dfg, :l0, ContinuousScalar, labels = [:LANDMARK], solvable=1)
+```@docs
+getLabel
 ```
 
-### Creating Factors with IIF
+#### Timestamps
 
-Similarly to variables, it is recommended that users start with the IIF implementation of the `addFactor!` functions to create factors. To create the factors, you provide the following parameters:
-- The graph the variable is being added to
-- The labels for the variables that the factor is linking
-- The factor function (which is a subtype of )
+Each variable or factor can have a timestamp associated with it.
 
-Additionally, the solvable flag is also set to indicate that the factor can be used in solving graphs.
+```@docs
+getTimestamp
+setTimestamp!
+```
 
-**NOTE:** Every graph requires a prior for it to be solvable, so it is a good practice to make sure one is added (generally by adding to the first variable in the graph).
+#### Tags
 
-Four factors are added: a prior, a linear conditional relationship with a normal distribution between x0 and x1, and a pair of linear conditional relationships between each pose and the landmark.
+Tags are a set of symbols that contain identifiers for the variable or factor.
 
-```julia
-prior = addFactor!(dfg, [:x0], Prior(Normal(0,1)))
-f1 = addFactor!(dfg, [:x0; :x1], LinearConditional(Normal(50.0,2.0)), solvable=1)
-f1 = addFactor!(dfg, [:l0; :x0], LinearConditional(Normal(40.0,5.0)), solvable=1)
-f1 = addFactor!(dfg, [:l0; :x1], LinearConditional(Normal(-10.0,5.0)), solvable=1)
+```@docs
+addTag
+mergeTags!
+deleteTags!
 ```
 
-The produced factor graph is:
+### Solvable
 
-![imgs/initialgraph.jpg](imgs/initialgraph.jpg)
+The solvable flag indicates whether the solver should make use of the variable or factor while solving the graph. This can be used to construct graphs in chunks while solving asynchronously, or for selectively solving portions of the graph.
 
-(For more information on producing plots of the graph, please refer to the
-[Drawing Graphs](DrawingGraphs.md) section).
+```@docs
+getSolvable
+setSolvable!
+```
 
-## Listing Variables and Factors
+### Variables
 
-Reading, updating, and deleting all use DFG functions (as opposed to adding,
-where using the IncrementalInference functions are recommended).
+#### Soft Type
 
-Each variable and factor is uniquely identified by its label. The list of
-variable and factor labels can be retrieved with the `ls`/`getVariableIds` and
-`lsf`/`getFactorIds` functions:
+The soft type is the underlying inference variable type, such as a Pose2.
 
 ```@docs
-getVariableIds
-ls
+getSofttype
 ```
 
-```@docs
-getFactorIds
-lsf
-```
+#### Packed Parametric Estimates
 
-To list all variables or factors (instead of just their labels), use the
-`getVariables` and `getFactors` functions:
+Solved graphs contain estimates for the variables, which are keyed by the solution (the default is saved as :default).
 
 ```@docs
-getVariables
-getFactors
 ```
 
-**NOTE**: `getNeighbors` is also worth mentioning at this point as it is a simple way to
-find the bigraph relationships. More information on this and other ways to
-retrieve filtered lists of variables/factors (an area that's currently WIP in
-DFG) can be found in [Traversing and Querying](TraversingAndQuerying.md).  
+#### Solver Data
 
-## Getting (Reading) Variables and Factors
+Solver data is used by IncrementalInference/RoME/Caesar solver to produce the above PPEs.
 
-Individual variables and factors can be retrieved from their labels using the following functions:
+Example of updating solver data:
 
-```@docs
-getVariable
+```julia
+# Add new VND of type ContinuousScalar to :x0
+# Could also do VariableNodeData(ContinuousScalar())
+vnd = VariableNodeData{ContinuousScalar}()
+addVariableSolverData!(dfg, :x0, vnd, :parametric)
+@show listVariableSolverData(dfg, :x0)
+# Get the data back - note that this is a reference to above.
+vndBack = getVariableSolverData(dfg, :x0, :parametric)
+# Delete it
+deleteVariableSolverData!(dfg, :x0, :parametric)
 ```
 
+Related Functions:
+
 ```@docs
-getFactor
+listVariableSolverData
+getVariableSolverData
+addVariableSolverData!
+updateVariableSolverData!
+deleteVariableSolverData!
 ```
 
-It is worth noting that `getVariable` allows a user to retrieve only a single
-solver entry, so that subsets of the solver data can be retrieved individually
-(say, in the case that there are many solutions). These can then be updated
-independently using the functions as discussed in the update section below.
+#### Small Data
 
-## Updating Variables and Factors
+#### Big Data
 
-Full variables and factors can be updated using the following functions:
+### Factors
 
-```@docs
-updateVariable!
-```
+## Graph-Related Data
 
-```@docs
-updateFactor!
-```
+DFG can store data in the graph itself (as opposed to inside graph elements).
+When you retrieve graphs from a database, this information is carried along. If
+you are working with an in-memory graph, the structure is flattened into the
+graph itself as `userData`, `robotData`, and `sessionData`.
 
-**NOTE**: Skeleton and summary variables are read-only. To perform updates you
-should use the full factors and variables.
+Graphs reside inside a hierarchy made up in the following way:
+- User1
+  - Robot1
+    - Session1 (the graph itself)
+- User2
+  - Robot2
+  - Robot3
+    - Session2
+    - Session3
 
-**NOTE**: `updateVariable`/`updateFactor` performs a complete update of the
-respective node. It's not a very efficient way to edit fine-grain detail. There
-are other methods to perform smaller in-place changes. This is discussed in
-more detail in [Data Structure](DataStructure.md).
+This data can be retrieved with the follow functions:
 
-## Deleting Variables and Factors
+```@docs
+getUserData
+getRobotData
+getSessionData
+```
 
-Variables and factors can be deleted using the following functions:
+It can be set using the following functions:
 
 ```@docs
-deleteVariable!
+setUserData!
+setRobotData!
+setSessionData!
 ```
 
-```@docs
-deleteFactor!
+Example of using graph-level data:
+
+```julia
+setUserData!(dfg, Dict(:a => "Hello"))
+getUserData(dfg)
 ```

src/CloudGraphsDFG/services/CGStructure.jl

@@ -296,23 +296,23 @@ function getUserData(dfg::CloudGraphsDFG)::Dict{Symbol, String}
     propVal = _getNodeProperty(dfg.neo4jInstance, [dfg.userId, "USER"], "data")
     return JSON2.read(String(base64decode(propVal)), Dict{Symbol, String})
 end
-function setUserData(dfg::CloudGraphsDFG, data::Dict{Symbol, String})::Bool
+function setUserData!(dfg::CloudGraphsDFG, data::Dict{Symbol, String})::Bool
     count = _setNodeProperty(dfg.neo4jInstance, [dfg.userId, "USER"], "data", base64encode(JSON2.write(data)))
     return count == 1
 end
 function getRobotData(dfg::CloudGraphsDFG)::Dict{Symbol, String}
     propVal = _getNodeProperty(dfg.neo4jInstance, [dfg.userId, dfg.robotId, "ROBOT"], "data")
     return JSON2.read(String(base64decode(propVal)), Dict{Symbol, String})
 end
-function setRobotData(dfg::CloudGraphsDFG, data::Dict{Symbol, String})::Bool
+function setRobotData!(dfg::CloudGraphsDFG, data::Dict{Symbol, String})::Bool
     count = _setNodeProperty(dfg.neo4jInstance, [dfg.userId, dfg.robotId, "ROBOT"], "data", base64encode(JSON2.write(data)))
     return count == 1
 end
 function getSessionData(dfg::CloudGraphsDFG)::Dict{Symbol, String}
     propVal = _getNodeProperty(dfg.neo4jInstance, [dfg.userId, dfg.robotId, dfg.sessionId, "SESSION"], "data")
     return JSON2.read(String(base64decode(propVal)), Dict{Symbol, String})
 end
-function setSessionData(dfg::CloudGraphsDFG, data::Dict{Symbol, String})::Bool
+function setSessionData!(dfg::CloudGraphsDFG, data::Dict{Symbol, String})::Bool
     count = _setNodeProperty(dfg.neo4jInstance, [dfg.userId, dfg.robotId, dfg.sessionId, "SESSION"], "data", base64encode(JSON2.write(data)))
     return count == 1
 end

src/DistributedFactorGraphs.jl

@@ -60,15 +60,18 @@ export getLabel, getTimestamp, setTimestamp!, getTags, setTags!
 export getMaxPPE, getMeanPPE, getSuggestedPPE, getVariablePPE, getPPE, getVariablePPEs, getPPEs #, getEstimates
 export getSofttype
 # Level 2
-export getSolverData, solverData, getData, getSolverDataDict, setSolverData!, getInternalId, smallData, setSmallData!, bigData
+export getData, getSolverData, getSolverDataDict, setSolverData!, getInternalId
+export listVariableSolverData, getVariableSolverData, addVariableSolverData!, updateVariableSolverData!, deleteVariableSolverData!
+
+export getSmallData, setSmallData!, bigData
 export addBigDataEntry!, getBigDataEntry, updateBigDataEntry!, deleteBigDataEntry!, getBigDataEntries, getBigDataKeys
 
 # Find a home
 export getVariableOrder
 
 # Services/AbstractDFG Exports
 export isInitialized, getFactorFunction, isVariable, isFactor
-export isSolvable, isSolveInProgress, getSolvable, setSolvable!, getSolveInProgress
+export isSolveInProgress, getSolvable, setSolvable!, getSolveInProgress
 export mergeUpdateVariableSolverData!, mergeUpdateGraphSolverData!
 
 # Solver (IIF) Exports
@@ -87,7 +90,7 @@ export MeanMaxPPE
 #--------
 export setSerializationModule!, getSerializationModule
 export getLabelDict, getDescription, setDescription, getAddHistory, getSolverParams, setSolverParams
-export getUserData, setUserData, getRobotData, setRobotData, getSessionData, setSessionData
+export getUserData, setUserData!, getRobotData, setRobotData!, getSessionData, setSessionData!
 
 # Not sure these are going to work everywhere, TODO implement in cloud?
 export updateUserData!, updateRobotData!, updateSessionData!, deleteUserData!, deleteRobotData!, deleteSessionData!