Skip to content

Incorrect Precision Loss in DECIMAL/NUMERIC Conversion #471

@dariocs

Description

@dariocs

Problem Description

When converting string literals to DECIMAL or NUMERIC types with precision > 16 digits, IBM i DB2 incorrectly rounds the values, losing precision beyond approximately 15-16 significant digits. This behavior occurs even when the target data type explicitly supports higher precision (e.g., DECIMAL(21,0)).

Expected Behavior

DECIMAL and NUMERIC types with specified precision should preserve exact values up to their declared precision limit (up to 31 digits on IBM i).

Actual Behavior

Values are rounded/truncated at approximately 15-16 digits during string-to-number conversion, regardless of the target precision specification.

Reproduction Steps

-- Test 1: DECIMAL function - INCORRECT ROUNDING
values(decimal('123456789012345678910', 21, 0));
-- Expected: 123456789012345678910
-- Actual: 123456789012345680000 (rounded)

-- Test 2: CAST to DECIMAL - INCORRECT ROUNDING  
values(cast('123456789012345678910' as decimal(21, 0)));
-- Expected: 123456789012345678910
-- Actual: 123456789012345680000 (rounded)

-- Test 3: CAST to NUMERIC - INCORRECT ROUNDING
values(cast('123456789012345678910' as numeric(21, 0)));
-- Expected: 123456789012345678910  
-- Actual: 123456789012345680000 (rounded)

-- Test 4: Comparison with working case (≤16 digits)
values(cast('1234567890123456' as decimal(16, 0)));
-- Expected: 1234567890123456
-- Actual: 1234567890123456 (correct)














<hr />

|Context|Version|
|-|-|
|Code for IBM i version|2.18.0|
|Visual Studio Code version|1.105.1|
|Operating System|win32_x64|


<details>
<summary>Active extensions</summary>

Account Microsoft (microsoft-authentication): 0.0.1
Autenticazione GitHub (github-authentication): 0.0.2
Azione per server pronto (debug-server-ready): 1.0.0
Browse Lite (browse-lite): 0.3.9
COBOL (cobol): 25.10.27
Code for IBM i Spooled Files (vscode-ibmi-userspooledfiles): 1.2.3
Code for IBM i Walkthroughs (vscode-ibmi-walkthroughs): 0.5.0
Collegamento automatico per debug di Node (debug-auto-launch): 1.0.0
Creazione estensione (extension-editing): 1.0.0
Db2 for IBM i (vscode-db2i): 1.15.1
ESLint (vscode-eslint): 3.0.16
Emmet (emmet): 1.0.0
Error Lens (errorlens): 3.26.0
Esegui merge del conflitto (merge-conflict): 1.0.0
Excel Viewer (gc-excelviewer): 4.2.64
Funzionalità dei linguaggi TypeScript e JavaScript (typescript-language-features): 1.0.0
Funzionalità del linguaggio JSON (json-language-features): 1.0.0
Funzionalità del linguaggio Markdown (markdown-language-features): 1.0.0
GIT (git): 1.0.0
Git Base (git-base): 1.0.0
GitHub (github): 0.0.1
GitHub Copilot (copilot): 1.388.0
GitHub Copilot Chat (copilot-chat): 0.32.5
IBM i Debug (ibmidebug): 3.0.2
IBM i Notebooks (vscode-ibmi-notebooks): 0.0.6
IBM i Project Explorer (vscode-ibmi-projectexplorer): 2.12.3
IBM i Testing (vscode-ibmi-testing): 1.2.3
Java (oracle-java): 24.1.2
Live Server (LiveServer): 5.7.9
Matematica Markdown (markdown-math): 1.0.0
Modifica della configurazione (configuration-editing): 1.0.0
Path Intellisense (path-intellisense): 2.10.0
RPGLE (vscode-rpgle): 0.33.3
Source Orbit (vscode-sourceorbit): 1.4.1
Suggerimento terminale per VS Code (terminal-suggest): 1.0.1
Supporto di npm per VS Code (npm): 1.0.1
TODO Highlight (vscode-todo-highlight): 1.0.5

</details>





<details>
<summary>Remote system</summary>

|Setting|Value|
|-|-|
|IBM i OS|V7R5M0|
|Tech Refresh|5|
|CCSID Origin|65535|
|Runtime CCSID|1144|
|Default CCSID|1144|
|SSHD CCSID|?|
|cqsh|true|
|SQL|Enabled
|Source dates|Enabled

### Enabled features

|/QOpenSys/pkgs/bin|/usr/bin|/QSYS.lib/ILEDITOR.lib|/QSYS.LIB|/QIBM/ProdData/IBMiDebugService/bin|/QOpenSys/QIBM/ProdData/JavaVM/jdk80|/QOpenSys/QIBM/ProdData/JavaVM/jdk11|/QOpenSys/QIBM/ProdData/JavaVM/jdk17|
|-|-|-|-|-|-|-|-|
|bash|attr|GETNEWLIBL.PGM|QZDFMDB2.PGM|startDebugService.sh|64bit|64bit|64bit|
|chsh|iconv|||||||
|find|setccsid|||||||
|git|uname|||||||
|grep||||||||
|ls||||||||
|md5sum||||||||
|pfgrep||||||||
|sort||||||||
|stat||||||||
|tar||||||||
|tn5250||||||||
</details>



<details>
<summary>Shell env</summary>

```bash
BUILDLIB=BFIT1778LB
CURLIB=xxx
EDITOR=vim
HOME=/home/xxx
HOST=
LIBLS=RPGUNIT QGPL QTEMP
LOGIN=bfmi1778
LOGNAME=bfmi1778
MAIL=/var/spool/mail/xxx
OLDPWD=/home/xxx
PASE_USRGRP_LIMITED=N
PATH=/QOpenSys/pkgs/bin:/QOpenSys/usr/bin:/usr/ccs/bin:/QOpenSys/usr/bin/X11:/usr/sbin:.:/usr/bin
PWD=/home/xxx
SHELL=/QOpenSys/pkgs/bin/bash
SHLVL=1

</details>



<details>
<summary>Variants</summary>

```json
{
  "american": "#@$",
  "local": "£§$",
  "qsysNameRegex": {}
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions