@@ -52,7 +52,7 @@ CONTENTS
52
52
53
53
- Varieties of memory barrier.
54
54
- What may not be assumed about memory barriers?
55
- - Data dependency barriers (historical).
55
+ - Address- dependency barriers (historical).
56
56
- Control dependencies.
57
57
- SMP barrier pairing.
58
58
- Examples of memory barrier sequences.
@@ -187,7 +187,7 @@ As a further example, consider this sequence of events:
187
187
B = 4; Q = P;
188
188
P = &B; D = *Q;
189
189
190
- There is an obvious data dependency here, as the value loaded into D depends on
190
+ There is an obvious address dependency here, as the value loaded into D depends on
191
191
the address retrieved from P by CPU 2. At the end of the sequence, any of the
192
192
following results are possible:
193
193
@@ -391,57 +391,61 @@ Memory barriers come in four basic varieties:
391
391
memory system as time progresses. All stores _before_ a write barrier
392
392
will occur _before_ all the stores after the write barrier.
393
393
394
- [!] Note that write barriers should normally be paired with read or data
395
- dependency barriers; see the "SMP barrier pairing" subsection.
394
+ [!] Note that write barriers should normally be paired with read or
395
+ address- dependency barriers; see the "SMP barrier pairing" subsection.
396
396
397
397
398
- (2) Data dependency barriers.
398
+ (2) Address- dependency barriers (historical) .
399
399
400
- A data dependency barrier is a weaker form of read barrier. In the case
400
+ An address- dependency barrier is a weaker form of read barrier. In the case
401
401
where two loads are performed such that the second depends on the result
402
402
of the first (eg: the first load retrieves the address to which the second
403
- load will be directed), a data dependency barrier would be required to
403
+ load will be directed), an address- dependency barrier would be required to
404
404
make sure that the target of the second load is updated after the address
405
405
obtained by the first load is accessed.
406
406
407
- A data dependency barrier is a partial ordering on interdependent loads
407
+ An address- dependency barrier is a partial ordering on interdependent loads
408
408
only; it is not required to have any effect on stores, independent loads
409
409
or overlapping loads.
410
410
411
411
As mentioned in (1), the other CPUs in the system can be viewed as
412
412
committing sequences of stores to the memory system that the CPU being
413
- considered can then perceive. A data dependency barrier issued by the CPU
413
+ considered can then perceive. An address- dependency barrier issued by the CPU
414
414
under consideration guarantees that for any load preceding it, if that
415
415
load touches one of a sequence of stores from another CPU, then by the
416
416
time the barrier completes, the effects of all the stores prior to that
417
- touched by the load will be perceptible to any loads issued after the data
417
+ touched by the load will be perceptible to any loads issued after the address-
418
418
dependency barrier.
419
419
420
420
See the "Examples of memory barrier sequences" subsection for diagrams
421
421
showing the ordering constraints.
422
422
423
- [!] Note that the first load really has to have a _data_ dependency and
423
+ [!] Note that the first load really has to have an _address_ dependency and
424
424
not a control dependency. If the address for the second load is dependent
425
425
on the first load, but the dependency is through a conditional rather than
426
426
actually loading the address itself, then it's a _control_ dependency and
427
427
a full read barrier or better is required. See the "Control dependencies"
428
428
subsection for more information.
429
429
430
- [!] Note that data dependency barriers should normally be paired with
430
+ [!] Note that address- dependency barriers should normally be paired with
431
431
write barriers; see the "SMP barrier pairing" subsection.
432
432
433
+ [!] Kernel release v5.9 removed kernel APIs for explicit address-
434
+ dependency barriers. Nowadays, APIs for marking loads from shared
435
+ variables such as READ_ONCE() and rcu_dereference() provide implicit
436
+ address-dependency barriers.
433
437
434
438
(3) Read (or load) memory barriers.
435
439
436
- A read barrier is a data dependency barrier plus a guarantee that all the
440
+ A read barrier is an address- dependency barrier plus a guarantee that all the
437
441
LOAD operations specified before the barrier will appear to happen before
438
442
all the LOAD operations specified after the barrier with respect to the
439
443
other components of the system.
440
444
441
445
A read barrier is a partial ordering on loads only; it is not required to
442
446
have any effect on stores.
443
447
444
- Read memory barriers imply data dependency barriers, and so can substitute
448
+ Read memory barriers imply address- dependency barriers, and so can substitute
445
449
for them.
446
450
447
451
[!] Note that read barriers should normally be paired with write barriers;
@@ -550,17 +554,21 @@ There are certain things that the Linux kernel memory barriers do not guarantee:
550
554
Documentation/core-api/dma-api.rst
551
555
552
556
553
- DATA DEPENDENCY BARRIERS (HISTORICAL)
554
- -------------------------------------
557
+ ADDRESS- DEPENDENCY BARRIERS (HISTORICAL)
558
+ ----------------------------------------
555
559
556
560
As of v4.15 of the Linux kernel, an smp_mb() was added to READ_ONCE() for
557
561
DEC Alpha, which means that about the only people who need to pay attention
558
562
to this section are those working on DEC Alpha architecture-specific code
559
563
and those working on READ_ONCE() itself. For those who need it, and for
560
564
those who are interested in the history, here is the story of
561
- data-dependency barriers.
565
+ address-dependency barriers.
566
+
567
+ [!] While address dependencies are observed in both load-to-load and
568
+ load-to-store relations, address-dependency barriers are not necessary
569
+ for load-to-store situations.
562
570
563
- The usage requirements of data dependency barriers are a little subtle, and
571
+ The requirement of address- dependency barriers is a little subtle, and
564
572
it's not always obvious that they're needed. To illustrate, consider the
565
573
following sequence of events:
566
574
@@ -570,10 +578,13 @@ following sequence of events:
570
578
B = 4;
571
579
<write barrier>
572
580
WRITE_ONCE(P, &B);
573
- Q = READ_ONCE (P);
581
+ Q = READ_ONCE_OLD (P);
574
582
D = *Q;
575
583
576
- There's a clear data dependency here, and it would seem that by the end of the
584
+ [!] READ_ONCE_OLD() corresponds to READ_ONCE() of pre-4.15 kernel, which
585
+ doesn't imply an address-dependency barrier.
586
+
587
+ There's a clear address dependency here, and it would seem that by the end of the
577
588
sequence, Q must be either &A or &B, and that:
578
589
579
590
(Q == &A) implies (D == 1)
@@ -588,8 +599,8 @@ While this may seem like a failure of coherency or causality maintenance, it
588
599
isn't, and this behaviour can be observed on certain real CPUs (such as the DEC
589
600
Alpha).
590
601
591
- To deal with this, a data dependency barrier or better must be inserted
592
- between the address load and the data load :
602
+ To deal with this, READ_ONCE() provides an implicit address-dependency
603
+ barrier since kernel release v4.15 :
593
604
594
605
CPU 1 CPU 2
595
606
=============== ===============
@@ -598,7 +609,7 @@ between the address load and the data load:
598
609
<write barrier>
599
610
WRITE_ONCE(P, &B);
600
611
Q = READ_ONCE(P);
601
- <data dependency barrier>
612
+ <implicit address- dependency barrier>
602
613
D = *Q;
603
614
604
615
This enforces the occurrence of one of the two implications, and prevents the
@@ -615,7 +626,7 @@ odd-numbered bank is idle, one can see the new value of the pointer P (&B),
615
626
but the old value of the variable B (2).
616
627
617
628
618
- A data -dependency barrier is not required to order dependent writes
629
+ An address -dependency barrier is not required to order dependent writes
619
630
because the CPUs that the Linux kernel supports don't do writes
620
631
until they are certain (1) that the write will actually happen, (2)
621
632
of the location of the write, and (3) of the value to be written.
@@ -629,12 +640,12 @@ break dependencies in a great many highly creative ways.
629
640
B = 4;
630
641
<write barrier>
631
642
WRITE_ONCE(P, &B);
632
- Q = READ_ONCE (P);
643
+ Q = READ_ONCE_OLD (P);
633
644
WRITE_ONCE(*Q, 5);
634
645
635
- Therefore, no data -dependency barrier is required to order the read into
646
+ Therefore, no address -dependency barrier is required to order the read into
636
647
Q with the store into *Q. In other words, this outcome is prohibited,
637
- even without a data -dependency barrier:
648
+ even without an implicit address -dependency barrier of modern READ_ONCE() :
638
649
639
650
(Q == &B) && (B == 4)
640
651
@@ -645,12 +656,12 @@ can be used to record rare error conditions and the like, and the CPUs'
645
656
naturally occurring ordering prevents such records from being lost.
646
657
647
658
648
- Note well that the ordering provided by a data dependency is local to
659
+ Note well that the ordering provided by an address dependency is local to
649
660
the CPU containing it. See the section on "Multicopy atomicity" for
650
661
more information.
651
662
652
663
653
- The data dependency barrier is very important to the RCU system,
664
+ The address- dependency barrier is very important to the RCU system,
654
665
for example. See rcu_assign_pointer() and rcu_dereference() in
655
666
include/linux/rcupdate.h. This permits the current target of an RCU'd
656
667
pointer to be replaced with a new modified target, without the replacement
@@ -667,16 +678,17 @@ not understand them. The purpose of this section is to help you prevent
667
678
the compiler's ignorance from breaking your code.
668
679
669
680
A load-load control dependency requires a full read memory barrier, not
670
- simply a data dependency barrier to make it work correctly. Consider the
681
+ simply an (implicit) address- dependency barrier to make it work correctly. Consider the
671
682
following bit of code:
672
683
673
684
q = READ_ONCE(a);
685
+ <implicit address-dependency barrier>
674
686
if (q) {
675
- <data dependency barrier> /* BUG: No data dependency!!! */
687
+ /* BUG: No address dependency!!! */
676
688
p = READ_ONCE(b);
677
689
}
678
690
679
- This will not have the desired effect because there is no actual data
691
+ This will not have the desired effect because there is no actual address
680
692
dependency, but rather a control dependency that the CPU may short-circuit
681
693
by attempting to predict the outcome in advance, so that other CPUs see
682
694
the load from b as having happened before the load from a. In such a
@@ -927,9 +939,9 @@ General barriers pair with each other, though they also pair with most
927
939
other types of barriers, albeit without multicopy atomicity. An acquire
928
940
barrier pairs with a release barrier, but both may also pair with other
929
941
barriers, including of course general barriers. A write barrier pairs
930
- with a data dependency barrier, a control dependency, an acquire barrier,
942
+ with an address- dependency barrier, a control dependency, an acquire barrier,
931
943
a release barrier, a read barrier, or a general barrier. Similarly a
932
- read barrier, control dependency, or a data dependency barrier pairs
944
+ read barrier, control dependency, or an address- dependency barrier pairs
933
945
with a write barrier, an acquire barrier, a release barrier, or a
934
946
general barrier:
935
947
948
960
a = 1;
949
961
<write barrier>
950
962
WRITE_ONCE(b, &a); x = READ_ONCE(b);
951
- <data dependency barrier>
963
+ <implicit address- dependency barrier>
952
964
y = *x;
953
965
954
966
Or even:
@@ -968,7 +980,7 @@ Basically, the read barrier always has to be there, even though it can be of
968
980
the "weaker" type.
969
981
970
982
[!] Note that the stores before the write barrier would normally be expected to
971
- match the loads after the read barrier or the data dependency barrier, and vice
983
+ match the loads after the read barrier or the address- dependency barrier, and vice
972
984
versa:
973
985
974
986
CPU 1 CPU 2
@@ -1021,7 +1033,7 @@ STORE B, STORE C } all occurring before the unordered set of { STORE D, STORE E
1021
1033
V
1022
1034
1023
1035
1024
- Secondly, data dependency barriers act as partial orderings on data -dependent
1036
+ Secondly, address- dependency barriers act as partial orderings on address -dependent
1025
1037
loads. Consider the following sequence of events:
1026
1038
1027
1039
CPU 1 CPU 2
@@ -1067,7 +1079,7 @@ effectively random order, despite the write barrier issued by CPU 1:
1067
1079
In the above example, CPU 2 perceives that B is 7, despite the load of *C
1068
1080
(which would be B) coming after the LOAD of C.
1069
1081
1070
- If, however, a data dependency barrier were to be placed between the load of C
1082
+ If, however, an address- dependency barrier were to be placed between the load of C
1071
1083
and the load of *C (ie: B) on CPU 2:
1072
1084
1073
1085
CPU 1 CPU 2
@@ -1078,7 +1090,7 @@ and the load of *C (ie: B) on CPU 2:
1078
1090
<write barrier>
1079
1091
STORE C = &B LOAD X
1080
1092
STORE D = 4 LOAD C (gets &B)
1081
- <data dependency barrier>
1093
+ <address- dependency barrier>
1082
1094
LOAD *C (reads B)
1083
1095
1084
1096
then the following will occur:
@@ -1101,7 +1113,7 @@ then the following will occur:
1101
1113
| +-------+ | |
1102
1114
| | X->9 |------>| |
1103
1115
| +-------+ | |
1104
- Makes sure all effects ---> \ ddddddddddddddddd | |
1116
+ Makes sure all effects ---> \ aaaaaaaaaaaaaaaaa | |
1105
1117
prior to the store of C \ +-------+ | |
1106
1118
are perceptible to ----->| B->2 |------>| |
1107
1119
subsequent loads +-------+ | |
@@ -1292,7 +1304,7 @@ Which might appear as this:
1292
1304
LOAD with immediate effect : : +-------+
1293
1305
1294
1306
1295
- Placing a read barrier or a data dependency barrier just before the second
1307
+ Placing a read barrier or an address- dependency barrier just before the second
1296
1308
load:
1297
1309
1298
1310
CPU 1 CPU 2
@@ -1816,20 +1828,20 @@ which may then reorder things however it wishes.
1816
1828
CPU MEMORY BARRIERS
1817
1829
-------------------
1818
1830
1819
- The Linux kernel has eight basic CPU memory barriers:
1831
+ The Linux kernel has seven basic CPU memory barriers:
1820
1832
1821
- TYPE MANDATORY SMP CONDITIONAL
1822
- =============== ======================= ============ ===============
1823
- GENERAL mb() smp_mb()
1824
- WRITE wmb() smp_wmb()
1825
- READ rmb() smp_rmb()
1826
- DATA DEPENDENCY READ_ONCE()
1833
+ TYPE MANDATORY SMP CONDITIONAL
1834
+ ======================= =============== ===============
1835
+ GENERAL mb() smp_mb()
1836
+ WRITE wmb() smp_wmb()
1837
+ READ rmb() smp_rmb()
1838
+ ADDRESS DEPENDENCY READ_ONCE()
1827
1839
1828
1840
1829
- All memory barriers except the data dependency barriers imply a compiler
1830
- barrier. Data dependencies do not impose any additional compiler ordering.
1841
+ All memory barriers except the address- dependency barriers imply a compiler
1842
+ barrier. Address dependencies do not impose any additional compiler ordering.
1831
1843
1832
- Aside: In the case of data dependencies, the compiler would be expected
1844
+ Aside: In the case of address dependencies, the compiler would be expected
1833
1845
to issue the loads in the correct order (eg. `a[b]` would have to load
1834
1846
the value of b before loading a[b]), however there is no guarantee in
1835
1847
the C specification that the compiler may not speculate the value of b
@@ -2889,7 +2901,7 @@ AND THEN THERE'S THE ALPHA
2889
2901
The DEC Alpha CPU is one of the most relaxed CPUs there is. Not only that,
2890
2902
some versions of the Alpha CPU have a split data cache, permitting them to have
2891
2903
two semantically-related cache lines updated at separate times. This is where
2892
- the data dependency barrier really becomes necessary as this synchronises both
2904
+ the address- dependency barrier really becomes necessary as this synchronises both
2893
2905
caches with the memory coherence system, thus making it seem like pointer
2894
2906
changes vs new data occur in the right order.
2895
2907
0 commit comments