@@ -52,7 +52,7 @@ CONTENTS
52
52
53
53
- Varieties of memory barrier.
54
54
- What may not be assumed about memory barriers?
55
- - Data dependency barriers (historical).
55
+ - Address- dependency barriers (historical).
56
56
- Control dependencies.
57
57
- SMP barrier pairing.
58
58
- Examples of memory barrier sequences.
@@ -187,9 +187,9 @@ As a further example, consider this sequence of events:
187
187
B = 4; Q = P;
188
188
P = &B; D = *Q;
189
189
190
- There is an obvious data dependency here, as the value loaded into D depends on
191
- the address retrieved from P by CPU 2. At the end of the sequence, any of the
192
- following results are possible:
190
+ There is an obvious address dependency here, as the value loaded into D depends
191
+ on the address retrieved from P by CPU 2. At the end of the sequence, any of
192
+ the following results are possible:
193
193
194
194
(Q == &A) and (D == 1)
195
195
(Q == &B) and (D == 2)
@@ -391,58 +391,62 @@ Memory barriers come in four basic varieties:
391
391
memory system as time progresses. All stores _before_ a write barrier
392
392
will occur _before_ all the stores after the write barrier.
393
393
394
- [!] Note that write barriers should normally be paired with read or data
395
- dependency barriers; see the "SMP barrier pairing" subsection.
394
+ [!] Note that write barriers should normally be paired with read or
395
+ address- dependency barriers; see the "SMP barrier pairing" subsection.
396
396
397
397
398
- (2) Data dependency barriers.
398
+ (2) Address- dependency barriers (historical) .
399
399
400
- A data dependency barrier is a weaker form of read barrier. In the case
401
- where two loads are performed such that the second depends on the result
402
- of the first (eg: the first load retrieves the address to which the second
403
- load will be directed), a data dependency barrier would be required to
404
- make sure that the target of the second load is updated after the address
405
- obtained by the first load is accessed.
400
+ An address- dependency barrier is a weaker form of read barrier. In the
401
+ case where two loads are performed such that the second depends on the
402
+ result of the first (eg: the first load retrieves the address to which
403
+ the second load will be directed), an address- dependency barrier would
404
+ be required to make sure that the target of the second load is updated
405
+ after the address obtained by the first load is accessed.
406
406
407
- A data dependency barrier is a partial ordering on interdependent loads
408
- only; it is not required to have any effect on stores, independent loads
409
- or overlapping loads.
407
+ An address- dependency barrier is a partial ordering on interdependent
408
+ loads only; it is not required to have any effect on stores, independent
409
+ loads or overlapping loads.
410
410
411
411
As mentioned in (1), the other CPUs in the system can be viewed as
412
412
committing sequences of stores to the memory system that the CPU being
413
- considered can then perceive. A data dependency barrier issued by the CPU
414
- under consideration guarantees that for any load preceding it, if that
415
- load touches one of a sequence of stores from another CPU, then by the
416
- time the barrier completes, the effects of all the stores prior to that
417
- touched by the load will be perceptible to any loads issued after the data
418
- dependency barrier.
413
+ considered can then perceive. An address- dependency barrier issued by
414
+ the CPU under consideration guarantees that for any load preceding it,
415
+ if that load touches one of a sequence of stores from another CPU, then
416
+ by the time the barrier completes, the effects of all the stores prior to
417
+ that touched by the load will be perceptible to any loads issued after
418
+ the address- dependency barrier.
419
419
420
420
See the "Examples of memory barrier sequences" subsection for diagrams
421
421
showing the ordering constraints.
422
422
423
- [!] Note that the first load really has to have a _data_ dependency and
423
+ [!] Note that the first load really has to have an _address_ dependency and
424
424
not a control dependency. If the address for the second load is dependent
425
425
on the first load, but the dependency is through a conditional rather than
426
426
actually loading the address itself, then it's a _control_ dependency and
427
427
a full read barrier or better is required. See the "Control dependencies"
428
428
subsection for more information.
429
429
430
- [!] Note that data dependency barriers should normally be paired with
430
+ [!] Note that address- dependency barriers should normally be paired with
431
431
write barriers; see the "SMP barrier pairing" subsection.
432
432
433
+ [!] Kernel release v5.9 removed kernel APIs for explicit address-
434
+ dependency barriers. Nowadays, APIs for marking loads from shared
435
+ variables such as READ_ONCE() and rcu_dereference() provide implicit
436
+ address-dependency barriers.
433
437
434
438
(3) Read (or load) memory barriers.
435
439
436
- A read barrier is a data dependency barrier plus a guarantee that all the
437
- LOAD operations specified before the barrier will appear to happen before
438
- all the LOAD operations specified after the barrier with respect to the
439
- other components of the system.
440
+ A read barrier is an address- dependency barrier plus a guarantee that all
441
+ the LOAD operations specified before the barrier will appear to happen
442
+ before all the LOAD operations specified after the barrier with respect to
443
+ the other components of the system.
440
444
441
445
A read barrier is a partial ordering on loads only; it is not required to
442
446
have any effect on stores.
443
447
444
- Read memory barriers imply data dependency barriers, and so can substitute
445
- for them.
448
+ Read memory barriers imply address- dependency barriers, and so can
449
+ substitute for them.
446
450
447
451
[!] Note that read barriers should normally be paired with write barriers;
448
452
see the "SMP barrier pairing" subsection.
@@ -550,17 +554,21 @@ There are certain things that the Linux kernel memory barriers do not guarantee:
550
554
Documentation/core-api/dma-api.rst
551
555
552
556
553
- DATA DEPENDENCY BARRIERS (HISTORICAL)
554
- -------------------------------------
557
+ ADDRESS- DEPENDENCY BARRIERS (HISTORICAL)
558
+ ----------------------------------------
555
559
556
560
As of v4.15 of the Linux kernel, an smp_mb() was added to READ_ONCE() for
557
561
DEC Alpha, which means that about the only people who need to pay attention
558
562
to this section are those working on DEC Alpha architecture-specific code
559
563
and those working on READ_ONCE() itself. For those who need it, and for
560
564
those who are interested in the history, here is the story of
561
- data-dependency barriers.
565
+ address-dependency barriers.
566
+
567
+ [!] While address dependencies are observed in both load-to-load and
568
+ load-to-store relations, address-dependency barriers are not necessary
569
+ for load-to-store situations.
562
570
563
- The usage requirements of data dependency barriers are a little subtle, and
571
+ The requirement of address- dependency barriers is a little subtle, and
564
572
it's not always obvious that they're needed. To illustrate, consider the
565
573
following sequence of events:
566
574
@@ -570,11 +578,14 @@ following sequence of events:
570
578
B = 4;
571
579
<write barrier>
572
580
WRITE_ONCE(P, &B);
573
- Q = READ_ONCE (P);
581
+ Q = READ_ONCE_OLD (P);
574
582
D = *Q;
575
583
576
- There's a clear data dependency here, and it would seem that by the end of the
577
- sequence, Q must be either &A or &B, and that:
584
+ [!] READ_ONCE_OLD() corresponds to READ_ONCE() of pre-4.15 kernel, which
585
+ doesn't imply an address-dependency barrier.
586
+
587
+ There's a clear address dependency here, and it would seem that by the end of
588
+ the sequence, Q must be either &A or &B, and that:
578
589
579
590
(Q == &A) implies (D == 1)
580
591
(Q == &B) implies (D == 4)
@@ -588,8 +599,8 @@ While this may seem like a failure of coherency or causality maintenance, it
588
599
isn't, and this behaviour can be observed on certain real CPUs (such as the DEC
589
600
Alpha).
590
601
591
- To deal with this, a data dependency barrier or better must be inserted
592
- between the address load and the data load :
602
+ To deal with this, READ_ONCE() provides an implicit address-dependency barrier
603
+ since kernel release v4.15 :
593
604
594
605
CPU 1 CPU 2
595
606
=============== ===============
@@ -598,7 +609,7 @@ between the address load and the data load:
598
609
<write barrier>
599
610
WRITE_ONCE(P, &B);
600
611
Q = READ_ONCE(P);
601
- <data dependency barrier>
612
+ <implicit address- dependency barrier>
602
613
D = *Q;
603
614
604
615
This enforces the occurrence of one of the two implications, and prevents the
@@ -615,26 +626,26 @@ odd-numbered bank is idle, one can see the new value of the pointer P (&B),
615
626
but the old value of the variable B (2).
616
627
617
628
618
- A data -dependency barrier is not required to order dependent writes
619
- because the CPUs that the Linux kernel supports don't do writes
620
- until they are certain (1) that the write will actually happen, (2)
621
- of the location of the write, and (3) of the value to be written.
629
+ An address -dependency barrier is not required to order dependent writes
630
+ because the CPUs that the Linux kernel supports don't do writes until they
631
+ are certain (1) that the write will actually happen, (2) of the location of
632
+ the write, and (3) of the value to be written.
622
633
But please carefully read the "CONTROL DEPENDENCIES" section and the
623
- Documentation/RCU/rcu_dereference.rst file: The compiler can and does
624
- break dependencies in a great many highly creative ways.
634
+ Documentation/RCU/rcu_dereference.rst file: The compiler can and does break
635
+ dependencies in a great many highly creative ways.
625
636
626
637
CPU 1 CPU 2
627
638
=============== ===============
628
639
{ A == 1, B == 2, C = 3, P == &A, Q == &C }
629
640
B = 4;
630
641
<write barrier>
631
642
WRITE_ONCE(P, &B);
632
- Q = READ_ONCE (P);
643
+ Q = READ_ONCE_OLD (P);
633
644
WRITE_ONCE(*Q, 5);
634
645
635
- Therefore, no data -dependency barrier is required to order the read into
646
+ Therefore, no address -dependency barrier is required to order the read into
636
647
Q with the store into *Q. In other words, this outcome is prohibited,
637
- even without a data -dependency barrier:
648
+ even without an implicit address -dependency barrier of modern READ_ONCE() :
638
649
639
650
(Q == &B) && (B == 4)
640
651
@@ -645,12 +656,12 @@ can be used to record rare error conditions and the like, and the CPUs'
645
656
naturally occurring ordering prevents such records from being lost.
646
657
647
658
648
- Note well that the ordering provided by a data dependency is local to
659
+ Note well that the ordering provided by an address dependency is local to
649
660
the CPU containing it. See the section on "Multicopy atomicity" for
650
661
more information.
651
662
652
663
653
- The data dependency barrier is very important to the RCU system,
664
+ The address- dependency barrier is very important to the RCU system,
654
665
for example. See rcu_assign_pointer() and rcu_dereference() in
655
666
include/linux/rcupdate.h. This permits the current target of an RCU'd
656
667
pointer to be replaced with a new modified target, without the replacement
@@ -667,20 +678,21 @@ not understand them. The purpose of this section is to help you prevent
667
678
the compiler's ignorance from breaking your code.
668
679
669
680
A load-load control dependency requires a full read memory barrier, not
670
- simply a data dependency barrier to make it work correctly. Consider the
671
- following bit of code:
681
+ simply an (implicit) address- dependency barrier to make it work correctly.
682
+ Consider the following bit of code:
672
683
673
684
q = READ_ONCE(a);
685
+ <implicit address-dependency barrier>
674
686
if (q) {
675
- <data dependency barrier> /* BUG: No data dependency!!! */
687
+ /* BUG: No address dependency!!! */
676
688
p = READ_ONCE(b);
677
689
}
678
690
679
- This will not have the desired effect because there is no actual data
691
+ This will not have the desired effect because there is no actual address
680
692
dependency, but rather a control dependency that the CPU may short-circuit
681
693
by attempting to predict the outcome in advance, so that other CPUs see
682
- the load from b as having happened before the load from a. In such a
683
- case what's actually required is:
694
+ the load from b as having happened before the load from a. In such a case
695
+ what's actually required is:
684
696
685
697
q = READ_ONCE(a);
686
698
if (q) {
@@ -927,9 +939,9 @@ General barriers pair with each other, though they also pair with most
927
939
other types of barriers, albeit without multicopy atomicity. An acquire
928
940
barrier pairs with a release barrier, but both may also pair with other
929
941
barriers, including of course general barriers. A write barrier pairs
930
- with a data dependency barrier, a control dependency, an acquire barrier,
942
+ with an address- dependency barrier, a control dependency, an acquire barrier,
931
943
a release barrier, a read barrier, or a general barrier. Similarly a
932
- read barrier, control dependency, or a data dependency barrier pairs
944
+ read barrier, control dependency, or an address- dependency barrier pairs
933
945
with a write barrier, an acquire barrier, a release barrier, or a
934
946
general barrier:
935
947
948
960
a = 1;
949
961
<write barrier>
950
962
WRITE_ONCE(b, &a); x = READ_ONCE(b);
951
- <data dependency barrier>
963
+ <implicit address- dependency barrier>
952
964
y = *x;
953
965
954
966
Or even:
@@ -968,8 +980,8 @@ Basically, the read barrier always has to be there, even though it can be of
968
980
the "weaker" type.
969
981
970
982
[!] Note that the stores before the write barrier would normally be expected to
971
- match the loads after the read barrier or the data dependency barrier, and vice
972
- versa:
983
+ match the loads after the read barrier or the address- dependency barrier, and
984
+ vice versa:
973
985
974
986
CPU 1 CPU 2
975
987
=================== ===================
@@ -1021,8 +1033,8 @@ STORE B, STORE C } all occurring before the unordered set of { STORE D, STORE E
1021
1033
V
1022
1034
1023
1035
1024
- Secondly, data dependency barriers act as partial orderings on data-dependent
1025
- loads. Consider the following sequence of events:
1036
+ Secondly, address- dependency barriers act as partial orderings on address-
1037
+ dependent loads. Consider the following sequence of events:
1026
1038
1027
1039
CPU 1 CPU 2
1028
1040
======================= =======================
@@ -1067,8 +1079,8 @@ effectively random order, despite the write barrier issued by CPU 1:
1067
1079
In the above example, CPU 2 perceives that B is 7, despite the load of *C
1068
1080
(which would be B) coming after the LOAD of C.
1069
1081
1070
- If, however, a data dependency barrier were to be placed between the load of C
1071
- and the load of *C (ie: B) on CPU 2:
1082
+ If, however, an address- dependency barrier were to be placed between the load
1083
+ of C and the load of *C (ie: B) on CPU 2:
1072
1084
1073
1085
CPU 1 CPU 2
1074
1086
======================= =======================
@@ -1078,7 +1090,7 @@ and the load of *C (ie: B) on CPU 2:
1078
1090
<write barrier>
1079
1091
STORE C = &B LOAD X
1080
1092
STORE D = 4 LOAD C (gets &B)
1081
- <data dependency barrier>
1093
+ <address- dependency barrier>
1082
1094
LOAD *C (reads B)
1083
1095
1084
1096
then the following will occur:
@@ -1101,7 +1113,7 @@ then the following will occur:
1101
1113
| +-------+ | |
1102
1114
| | X->9 |------>| |
1103
1115
| +-------+ | |
1104
- Makes sure all effects ---> \ ddddddddddddddddd | |
1116
+ Makes sure all effects ---> \ aaaaaaaaaaaaaaaaa | |
1105
1117
prior to the store of C \ +-------+ | |
1106
1118
are perceptible to ----->| B->2 |------>| |
1107
1119
subsequent loads +-------+ | |
@@ -1292,7 +1304,7 @@ Which might appear as this:
1292
1304
LOAD with immediate effect : : +-------+
1293
1305
1294
1306
1295
- Placing a read barrier or a data dependency barrier just before the second
1307
+ Placing a read barrier or an address- dependency barrier just before the second
1296
1308
load:
1297
1309
1298
1310
CPU 1 CPU 2
@@ -1816,20 +1828,20 @@ which may then reorder things however it wishes.
1816
1828
CPU MEMORY BARRIERS
1817
1829
-------------------
1818
1830
1819
- The Linux kernel has eight basic CPU memory barriers:
1831
+ The Linux kernel has seven basic CPU memory barriers:
1820
1832
1821
- TYPE MANDATORY SMP CONDITIONAL
1822
- =============== ======================= ============ ===============
1823
- GENERAL mb() smp_mb()
1824
- WRITE wmb() smp_wmb()
1825
- READ rmb() smp_rmb()
1826
- DATA DEPENDENCY READ_ONCE()
1833
+ TYPE MANDATORY SMP CONDITIONAL
1834
+ ======================= =============== ===============
1835
+ GENERAL mb() smp_mb()
1836
+ WRITE wmb() smp_wmb()
1837
+ READ rmb() smp_rmb()
1838
+ ADDRESS DEPENDENCY READ_ONCE()
1827
1839
1828
1840
1829
- All memory barriers except the data dependency barriers imply a compiler
1830
- barrier. Data dependencies do not impose any additional compiler ordering.
1841
+ All memory barriers except the address- dependency barriers imply a compiler
1842
+ barrier. Address dependencies do not impose any additional compiler ordering.
1831
1843
1832
- Aside: In the case of data dependencies, the compiler would be expected
1844
+ Aside: In the case of address dependencies, the compiler would be expected
1833
1845
to issue the loads in the correct order (eg. `a[b]` would have to load
1834
1846
the value of b before loading a[b]), however there is no guarantee in
1835
1847
the C specification that the compiler may not speculate the value of b
@@ -2749,7 +2761,8 @@ is discarded from the CPU's cache and reloaded. To deal with this, the
2749
2761
appropriate part of the kernel must invalidate the overlapping bits of the
2750
2762
cache on each CPU.
2751
2763
2752
- See Documentation/core-api/cachetlb.rst for more information on cache management.
2764
+ See Documentation/core-api/cachetlb.rst for more information on cache
2765
+ management.
2753
2766
2754
2767
2755
2768
CACHE COHERENCY VS MMIO
@@ -2889,8 +2902,8 @@ AND THEN THERE'S THE ALPHA
2889
2902
The DEC Alpha CPU is one of the most relaxed CPUs there is. Not only that,
2890
2903
some versions of the Alpha CPU have a split data cache, permitting them to have
2891
2904
two semantically-related cache lines updated at separate times. This is where
2892
- the data dependency barrier really becomes necessary as this synchronises both
2893
- caches with the memory coherence system, thus making it seem like pointer
2905
+ the address- dependency barrier really becomes necessary as this synchronises
2906
+ both caches with the memory coherence system, thus making it seem like pointer
2894
2907
changes vs new data occur in the right order.
2895
2908
2896
2909
The Alpha defines the Linux kernel's memory model, although as of v4.15
0 commit comments