Adapting tab mode to file content not working

[fleutot]

The documentation made me expect dtrt-indent to adapt indent-tabs-mode to "the right thing":

;; `indent-tabs-mode' Setting
;;
;; For determining hard vs. soft tabs, dtrt-indent counts the number of
;; lines out of the eligible lines in the fixed segment that are
;; indented using hard tabs, and the number of lines indented using
;; spaces.  If either count is significantly higher than the other count,
;; `indent-tabs-mode' will be modified.

However, opening this file /tmp/asdf.sh with basic init file ~/tmp/basic-init.el does not seem to guess and modify to the right value of indent-tabs-mode.

---

Test material

/tmp/asdf.sh:

#! /usr/bin/env bash

function asdf()
{
    printf "asASDFASDF"
    echo "1234"
}

printf "qwerqwerqwer\n"
result=$(asdf)

if [ df ] ; then
    if [ er ] ; then
        wefwefew
    fi
fi

basic-init.el:

(require 'package)
(setq package-enable-at-startup nil)
(add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/"))
(add-to-list 'package-archives '("gnu" . "http://elpa.gnu.org/packages/"))
(add-to-list 'package-archives '("melpamilk" . "http://melpa.milkbox.net/packages/"))
(package-initialize)

(unless package-archive-contents
  (package-refresh-contents))
(unless (package-installed-p 'use-package)
  (package-install 'use-package))
(require 'use-package)

(defun my-prog-mode-hook ()
  (dtrt-indent-mode))

(use-package dtrt-indent
  :ensure t
  :config (dtrt-indent-mode t))

(require 'whitespace) ;; builtin
(global-set-key (kbd "C-c b") 'whitespace-mode)
(delete 'indentation whitespace-style) ; no warning about indent with spaces

---

Test result

Starting emacs with emacs -q -l ~/tmp/basic-init.el /tmp/asdf.sh, then activating whitespace mode with C-c b gives me this:

After going to the row with 8 space indent (wefwefew), deleting one space, pressing TAB, I get this:

Expected result: dtrt-indent-mode indents with spaces, since all other indentations are with spaces.

---

Debug data

Here is what I have found so far.

Result of dtrt-indent-diagnosis:

Guessing offset for /tmp/asdf.sh

Elapsed time for analysis: 0.001 seconds

Total relevant lines: 5 out of 17 (limit: 5000)

Histogram:

     4x   4 spaces
     1x   8 spaces

Analysis:

  offset 2 works for 100.00% of relevant lines, matching 2 distinct offsets - merged with offset 4 (0.00% deviation, limit 20.00%)
  offset 4 works for 100.00% of relevant lines, matching 2 distinct offsets - CONSIDERED
  offset 8 works for  20.00% of relevant lines, matching 1 distinct offsets - CONSIDERED
  offset 3 works for   0.00% of relevant lines, matching 0 distinct offsets - rejected: too few distinct matching offsets (1 required)
  offset 5 works for   0.00% of relevant lines, matching 0 distinct offsets - rejected: too few distinct matching offsets (1 required)
  offset 6 works for   0.00% of relevant lines, matching 0 distinct offsets - rejected: too few distinct matching offsets (1 required)
  offset 7 works for   0.00% of relevant lines, matching 0 distinct offsets - rejected: too few distinct matching offsets (1 required)

Summary:

  Best guess is offset 4 with 100.00% matching lines (80.00% required)
  Hard tab percentage: 20.00% (1 lines), -75.00% superior to soft tabs (threshold 300.00%)
  Soft tab percentage: 80.00% (4 lines), 300.00% superior to hard tabs (threshold 300.00%)

Conclusion:

  Guessed offset 4 with 100% confidence.
  Change indent-tab-setting: yes, to nil

It seems dtrt-indent-mode could correctly guess that indent-tab-setting (although there is no such variable? Is there? Wrong name? Shouldn't it be indent-tabs-mode?) should be nil.
But it could only hint the user that they probably want to set it to nil (the message seems to be a recommendation, although it is not obvious. At first glance, it looks like dtrt-indent actually made the change).

C-h v indent-tabs-mode is t, so it was not changed. According to the code comment I posted at the top, it should?

The embedded help I open from that emacs session says:

When dtrt-indent mode is enabled, the proper indentation offset
and ‘indent-tabs-mode’ will be guessed for newly opened files and
adjusted transparently.

package-list-packages tells me that what I have installed is this:

     Status: Installed in ‘dtrt-indent-20210423.745/’ (unsigned). Delete
    Version: 20210423.745
     Commit: 9714f2c5f1c9b7c21e732df8c15a870a88caba84
    Summary: Adapt to foreign indentation offsets
   Keywords: convenience files languages c 
Other versions: 20210423.745 (melpa), 20200430.1023 (melpamilk).

[jscheid]

@fleutot set dtrt-indent-run-after-smie and it should work.

@rrthomas when dtrt-indent doesn't run because SMIE is active (and dtrt-indent-run-after-smie is nil) there is no indication to users: dtrt-indent-diagnosis doesn't mention it, the mode-line lighter is displayed, and the minor mode is listed as active. This is clearly confusing.

The diagnosis should say something like "dtrt-indent doesn't run in this buffer because...".

As for the minor mode, I haven't been involved with its refactoring: do you think it would be better to check inside the globalized mode to decide whether or not it should be enabled for a given buffer?

[jscheid]

I might see if I can make time to improve this.

[rrthomas]

Thanks very much, @jscheid, for looking into this and other issues; also thanks @fleutot for a beautifully detailed report for this issue.

[fleutot]

And thank you @jscheid and @rrthomas for looking into this :) If any of you wants to earn free internet pointz, here is the question I wrote on StackExchange about this.

Incidentally, and this is a little embarrassing, I didn't load the package correctly in my minimal example above either:

(use-package dtrt-indent
  :ensure t
  :config (dtrt-indent-mode t))  ; This line is NOT going to load the module by itself

I did create a custom prog mode hook, but never actually loaded it. How silly.

This works:

(defun my-prog-mode-hook ()
  (dtrt-indent-mode))

(add-hook 'prog-mode-hook 'my-prog-mode-hook)

(use-package dtrt-indent
  :ensure t
  :config
  (setq dtrt-indent-run-after-smie t)
  )

I kind of expected dtrt to replace smie altoghether. But now it looks like it rather requires it?

I suppose there is a use for dtrt-indent-mode without dtrt-indent-run-after-smie, but I don't see what it could be? If there isn't, why not make it default?

[rrthomas]

As far as the interaction with SMIE goes (since I introduced that) the idea is that for modes that use SMIE, dtrt-indent should defer to SMIE. If you don't use dtrt-indent-run-after-smie, then dtrt-indent will still be used in non-SMIE modes.

[fleutot]

I don't understand "defer to SMIE". Do you mean that dtrt-mode does nothing by default if SMIE is in use? Even if dtrt-mode is active? And dtrt-indent-run-after-smie overrides that and makes it run anyway?

If that is the case, I understand @jscheid's comment better, it really is confusing having the mode active but no effect. Why not have dtrt-indent-run-after-smie on by default? Whoever activates the mode (at least locally) certainly wants it active?

[rrthomas]

I don't understand "defer to SMIE". Do you mean that dtrt-mode does nothing by default if SMIE is in use? Even if dtrt-mode is active? And dtrt-indent-run-after-smie overrides that and makes it run anyway?

That's right.

Why not have dtrt-indent-run-after-smie on by default? Whoever activates the mode (at least locally) certainly wants it active?

Indeed, if you're activating dtrt-indent-mode locally, then it may make sense to turn dtrt-indent-run-after-smie on. The default setting of off makes sense when you're just using dtrt-indent-global-mode. The idea is that SMIE is supposed to solve the same problem, and it's the "default" solution, so we don't want to interfere with it by default.

[apteryks]

I think I encountered a similar/problem confusion: I have (setq dtrt-indent-global-mode t) in my ~/.emacs, and when I enter the git/t/t9001-send-email.sh file, for example, this is what I see in *Messages*, with dtrt-indent-verbosity set to 3:

Setting up indent for shell type sh
Indentation variables are now local.
Indentation setup for shell type sh

It seems dtrt-indent-mode is not even turned on by the global mode as the dtrt-indent-mode variable is nil; I'm not sure why that is.

If I manually M-x dtrt-indent-mode, then I see it defers to smie, eventually calling (smie-config--guess (point-min) (point-max)) and applying that with (smie-config-local config), where the guessed config's value is ((8 :after "do" 4) (8 :after "{" 4) (0 :elem basic 4)). It seems SMIE doesn't concern itself with the tabs vs space question?

If confirmed, that'd be a reason enough to have dtrt-indent-run-after-smie to t by default, in my opinion, as I assume being able to infer a correct indent-tabs-mode value is one of the features expected from this package.

I also note that:

• SMIE appears undocumented in the Emacs manual (info "emacs").
• smie.el has twice the amount of code as dtrt-indent, while apparently offering less
• I had never heard of smie until I read the source of dtrt-indent.el

:-)

[apteryks]

Perhaps, since smie doesn't infer correct tabs vs space usage, the variable should be renamed to dtrt-indent-use-smie option that would default to nil, and documented along

Use Emacs' builtin SMIE package to guess indentation rules.
SMIE is only used if it has support for the buffer's mode. Note that contrary to dtrt-indent-mode, SMIE does not adjust indent-tabs-mode itself.

[rrthomas]

These days, SMIE seems to have been overtaken by tree-sitter as a basis for writing programming language modes, so I'd be more gung-ho about having dtrt-indent override it by default.

Also, many programmers and projects will have editorconfig or similar mechanisms set up for particular projects (I know I do!). So, dtrt-indent is mostly for users who, like me, want it to be easy to modify code in projects that don't have that sort of setup, and would rather get an intelligent guess that works much of the time than have to manually set things up every time.

However, I think that unless SMIE is worse at guessing than dtrt-indent, then using it by default is still a good idea.

Which makes me think that dtrt-indent should evaluate and potentially change indent-tabs-mode, even when SMIE is used.

This suggests that dtrt-indent-try-set-offset should be factored out into a) analysis, and b) two setting functions, one for indent and one for tabs vs spaces.

Then, post-SMIE, we run analysis + tabs & spaces if dtrt-indent-run-after-smie is nil, and the "full-fat" dtrt-indent-try-set-offset if it's t.

How does that sound?

[apteryks]

However, I think that unless SMIE is worse at guessing than dtrt-indent, then using it by default is still a good idea.

I don't know; if it's going to be replaced by something newer and better (tree-sitters?), then if I were the maintainer of this package I wouldn't want to invest too much effort into refactoring the code base to support it. It's also not clear that it guesses better than dtrt-indent; it seems that'd need to be determined before deciding if using it by default is worthwhile.

The one report that led to supporting smie was #14:

Many Emacs 24.4 major modes are now using SMIE, a parser that improves automatic indentation and indentation based navigation. Many minor modes have switched to using it instead of the normal indentation variables, which breaks dtrt-indent in sh-mode for example.

It seems the issue there was not the accuracy of the guessing but the variables that dtrt-indent was using to set the indentation were no longer effective for the smie-relying modes? It's not clear to me :-).

But other than the uncertain effort/gain value of your proposal, it sounds like a good improvement/sound design idea. I'll be happy to put its implementation to the test!

[rrthomas]

However, I think that unless SMIE is worse at guessing than dtrt-indent, then using it by default is still a good idea.

I don't know; if it's going to be replaced by something newer and better (tree-sitters?), then if I were the maintainer of this package I wouldn't want to invest too much effort into refactoring the code base to support it. It's also not clear that it guesses better than dtrt-indent; it seems that'd need to be determined before deciding if using it by default is worthwhile.

The refactoring seems worth it anyway: dtrt-indent-try-set-offset is a big function that does 3 clearly separated things.

As you're probably aware, things take a long time to disappear from Emacs; I don't see SMIE going anywhere any time soon. Treesitter also seems to be widely loved & hated (and doesn't guess indentation, so needs no investment from dtrt-indent).

It seems the issue there was not the accuracy of the guessing but the variables that dtrt-indent was using to set the indentation were no longer effective for the smie-relying modes? It's not clear to me :-).

The problem IIRC was that SMIE and dtrt-indent ended up fighting to see who set the variables.

But other than the uncertain effort/gain value of your proposal, it sounds like a good improvement/sound design idea. I'll be happy to put its implementation to the test!

Great, I'll update the issue to make it an Official Idea. No guarantees I'll find time for it soon.

[rrthomas]

Summary of things that could be improved:

• When dtrt-indent doesn't run because SMIE is active (and dtrt-indent-run-after-smie is nil), mention this in dtrt-indent-diagnosis, ("dtrt-indent doesn't run in this buffer because...") and maybe check inside the globalized mode to decide whether or not it should be enabled for a given buffer.
• dtrt-indent should evaluate and potentially change indent-tabs-mode, even when SMIE is used.
• To achieve this, factor dtrt-indent-try-set-offset out into a) analysis, and b) two setting functions, one for indent and one for tabs vs spaces. Post-SMIE, if dtrt-indent-run-after-smie is set, run dtrt-indent-try-set-offset as at present; otherwise just run analysis and then set tabs mode.