rcu_dereference.txt 12 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367
  1. PROPER CARE AND FEEDING OF RETURN VALUES FROM rcu_dereference()
  2. Most of the time, you can use values from rcu_dereference() or one of
  3. the similar primitives without worries. Dereferencing (prefix "*"),
  4. field selection ("->"), assignment ("="), address-of ("&"), addition and
  5. subtraction of constants, and casts all work quite naturally and safely.
  6. It is nevertheless possible to get into trouble with other operations.
  7. Follow these rules to keep your RCU code working properly:
  8. o You must use one of the rcu_dereference() family of primitives
  9. to load an RCU-protected pointer, otherwise CONFIG_PROVE_RCU
  10. will complain. Worse yet, your code can see random memory-corruption
  11. bugs due to games that compilers and DEC Alpha can play.
  12. Without one of the rcu_dereference() primitives, compilers
  13. can reload the value, and won't your code have fun with two
  14. different values for a single pointer! Without rcu_dereference(),
  15. DEC Alpha can load a pointer, dereference that pointer, and
  16. return data preceding initialization that preceded the store of
  17. the pointer.
  18. In addition, the volatile cast in rcu_dereference() prevents the
  19. compiler from deducing the resulting pointer value. Please see
  20. the section entitled "EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH"
  21. for an example where the compiler can in fact deduce the exact
  22. value of the pointer, and thus cause misordering.
  23. o Avoid cancellation when using the "+" and "-" infix arithmetic
  24. operators. For example, for a given variable "x", avoid
  25. "(x-x)". There are similar arithmetic pitfalls from other
  26. arithmetic operators, such as "(x*0)", "(x/(x+1))" or "(x%1)".
  27. The compiler is within its rights to substitute zero for all of
  28. these expressions, so that subsequent accesses no longer depend
  29. on the rcu_dereference(), again possibly resulting in bugs due
  30. to misordering.
  31. Of course, if "p" is a pointer from rcu_dereference(), and "a"
  32. and "b" are integers that happen to be equal, the expression
  33. "p+a-b" is safe because its value still necessarily depends on
  34. the rcu_dereference(), thus maintaining proper ordering.
  35. o Avoid all-zero operands to the bitwise "&" operator, and
  36. similarly avoid all-ones operands to the bitwise "|" operator.
  37. If the compiler is able to deduce the value of such operands,
  38. it is within its rights to substitute the corresponding constant
  39. for the bitwise operation. Once again, this causes subsequent
  40. accesses to no longer depend on the rcu_dereference(), causing
  41. bugs due to misordering.
  42. Please note that single-bit operands to bitwise "&" can also
  43. be dangerous. At this point, the compiler knows that the
  44. resulting value can only take on one of two possible values.
  45. Therefore, a very small amount of additional information will
  46. allow the compiler to deduce the exact value, which again can
  47. result in misordering.
  48. o If you are using RCU to protect JITed functions, so that the
  49. "()" function-invocation operator is applied to a value obtained
  50. (directly or indirectly) from rcu_dereference(), you may need to
  51. interact directly with the hardware to flush instruction caches.
  52. This issue arises on some systems when a newly JITed function is
  53. using the same memory that was used by an earlier JITed function.
  54. o Do not use the results from the boolean "&&" and "||" when
  55. dereferencing. For example, the following (rather improbable)
  56. code is buggy:
  57. int *p;
  58. int *q;
  59. ...
  60. p = rcu_dereference(gp)
  61. q = &global_q;
  62. q += p != &oom_p1 && p != &oom_p2;
  63. r1 = *q; /* BUGGY!!! */
  64. The reason this is buggy is that "&&" and "||" are often compiled
  65. using branches. While weak-memory machines such as ARM or PowerPC
  66. do order stores after such branches, they can speculate loads,
  67. which can result in misordering bugs.
  68. o Do not use the results from relational operators ("==", "!=",
  69. ">", ">=", "<", or "<=") when dereferencing. For example,
  70. the following (quite strange) code is buggy:
  71. int *p;
  72. int *q;
  73. ...
  74. p = rcu_dereference(gp)
  75. q = &global_q;
  76. q += p > &oom_p;
  77. r1 = *q; /* BUGGY!!! */
  78. As before, the reason this is buggy is that relational operators
  79. are often compiled using branches. And as before, although
  80. weak-memory machines such as ARM or PowerPC do order stores
  81. after such branches, but can speculate loads, which can again
  82. result in misordering bugs.
  83. o Be very careful about comparing pointers obtained from
  84. rcu_dereference() against non-NULL values. As Linus Torvalds
  85. explained, if the two pointers are equal, the compiler could
  86. substitute the pointer you are comparing against for the pointer
  87. obtained from rcu_dereference(). For example:
  88. p = rcu_dereference(gp);
  89. if (p == &default_struct)
  90. do_default(p->a);
  91. Because the compiler now knows that the value of "p" is exactly
  92. the address of the variable "default_struct", it is free to
  93. transform this code into the following:
  94. p = rcu_dereference(gp);
  95. if (p == &default_struct)
  96. do_default(default_struct.a);
  97. On ARM and Power hardware, the load from "default_struct.a"
  98. can now be speculated, such that it might happen before the
  99. rcu_dereference(). This could result in bugs due to misordering.
  100. However, comparisons are OK in the following cases:
  101. o The comparison was against the NULL pointer. If the
  102. compiler knows that the pointer is NULL, you had better
  103. not be dereferencing it anyway. If the comparison is
  104. non-equal, the compiler is none the wiser. Therefore,
  105. it is safe to compare pointers from rcu_dereference()
  106. against NULL pointers.
  107. o The pointer is never dereferenced after being compared.
  108. Since there are no subsequent dereferences, the compiler
  109. cannot use anything it learned from the comparison
  110. to reorder the non-existent subsequent dereferences.
  111. This sort of comparison occurs frequently when scanning
  112. RCU-protected circular linked lists.
  113. o The comparison is against a pointer that references memory
  114. that was initialized "a long time ago." The reason
  115. this is safe is that even if misordering occurs, the
  116. misordering will not affect the accesses that follow
  117. the comparison. So exactly how long ago is "a long
  118. time ago"? Here are some possibilities:
  119. o Compile time.
  120. o Boot time.
  121. o Module-init time for module code.
  122. o Prior to kthread creation for kthread code.
  123. o During some prior acquisition of the lock that
  124. we now hold.
  125. o Before mod_timer() time for a timer handler.
  126. There are many other possibilities involving the Linux
  127. kernel's wide array of primitives that cause code to
  128. be invoked at a later time.
  129. o The pointer being compared against also came from
  130. rcu_dereference(). In this case, both pointers depend
  131. on one rcu_dereference() or another, so you get proper
  132. ordering either way.
  133. That said, this situation can make certain RCU usage
  134. bugs more likely to happen. Which can be a good thing,
  135. at least if they happen during testing. An example
  136. of such an RCU usage bug is shown in the section titled
  137. "EXAMPLE OF AMPLIFIED RCU-USAGE BUG".
  138. o All of the accesses following the comparison are stores,
  139. so that a control dependency preserves the needed ordering.
  140. That said, it is easy to get control dependencies wrong.
  141. Please see the "CONTROL DEPENDENCIES" section of
  142. Documentation/memory-barriers.txt for more details.
  143. o The pointers are not equal -and- the compiler does
  144. not have enough information to deduce the value of the
  145. pointer. Note that the volatile cast in rcu_dereference()
  146. will normally prevent the compiler from knowing too much.
  147. However, please note that if the compiler knows that the
  148. pointer takes on only one of two values, a not-equal
  149. comparison will provide exactly the information that the
  150. compiler needs to deduce the value of the pointer.
  151. o Disable any value-speculation optimizations that your compiler
  152. might provide, especially if you are making use of feedback-based
  153. optimizations that take data collected from prior runs. Such
  154. value-speculation optimizations reorder operations by design.
  155. There is one exception to this rule: Value-speculation
  156. optimizations that leverage the branch-prediction hardware are
  157. safe on strongly ordered systems (such as x86), but not on weakly
  158. ordered systems (such as ARM or Power). Choose your compiler
  159. command-line options wisely!
  160. EXAMPLE OF AMPLIFIED RCU-USAGE BUG
  161. Because updaters can run concurrently with RCU readers, RCU readers can
  162. see stale and/or inconsistent values. If RCU readers need fresh or
  163. consistent values, which they sometimes do, they need to take proper
  164. precautions. To see this, consider the following code fragment:
  165. struct foo {
  166. int a;
  167. int b;
  168. int c;
  169. };
  170. struct foo *gp1;
  171. struct foo *gp2;
  172. void updater(void)
  173. {
  174. struct foo *p;
  175. p = kmalloc(...);
  176. if (p == NULL)
  177. deal_with_it();
  178. p->a = 42; /* Each field in its own cache line. */
  179. p->b = 43;
  180. p->c = 44;
  181. rcu_assign_pointer(gp1, p);
  182. p->b = 143;
  183. p->c = 144;
  184. rcu_assign_pointer(gp2, p);
  185. }
  186. void reader(void)
  187. {
  188. struct foo *p;
  189. struct foo *q;
  190. int r1, r2;
  191. p = rcu_dereference(gp2);
  192. if (p == NULL)
  193. return;
  194. r1 = p->b; /* Guaranteed to get 143. */
  195. q = rcu_dereference(gp1); /* Guaranteed non-NULL. */
  196. if (p == q) {
  197. /* The compiler decides that q->c is same as p->c. */
  198. r2 = p->c; /* Could get 44 on weakly order system. */
  199. }
  200. do_something_with(r1, r2);
  201. }
  202. You might be surprised that the outcome (r1 == 143 && r2 == 44) is possible,
  203. but you should not be. After all, the updater might have been invoked
  204. a second time between the time reader() loaded into "r1" and the time
  205. that it loaded into "r2". The fact that this same result can occur due
  206. to some reordering from the compiler and CPUs is beside the point.
  207. But suppose that the reader needs a consistent view?
  208. Then one approach is to use locking, for example, as follows:
  209. struct foo {
  210. int a;
  211. int b;
  212. int c;
  213. spinlock_t lock;
  214. };
  215. struct foo *gp1;
  216. struct foo *gp2;
  217. void updater(void)
  218. {
  219. struct foo *p;
  220. p = kmalloc(...);
  221. if (p == NULL)
  222. deal_with_it();
  223. spin_lock(&p->lock);
  224. p->a = 42; /* Each field in its own cache line. */
  225. p->b = 43;
  226. p->c = 44;
  227. spin_unlock(&p->lock);
  228. rcu_assign_pointer(gp1, p);
  229. spin_lock(&p->lock);
  230. p->b = 143;
  231. p->c = 144;
  232. spin_unlock(&p->lock);
  233. rcu_assign_pointer(gp2, p);
  234. }
  235. void reader(void)
  236. {
  237. struct foo *p;
  238. struct foo *q;
  239. int r1, r2;
  240. p = rcu_dereference(gp2);
  241. if (p == NULL)
  242. return;
  243. spin_lock(&p->lock);
  244. r1 = p->b; /* Guaranteed to get 143. */
  245. q = rcu_dereference(gp1); /* Guaranteed non-NULL. */
  246. if (p == q) {
  247. /* The compiler decides that q->c is same as p->c. */
  248. r2 = p->c; /* Locking guarantees r2 == 144. */
  249. }
  250. spin_unlock(&p->lock);
  251. do_something_with(r1, r2);
  252. }
  253. As always, use the right tool for the job!
  254. EXAMPLE WHERE THE COMPILER KNOWS TOO MUCH
  255. If a pointer obtained from rcu_dereference() compares not-equal to some
  256. other pointer, the compiler normally has no clue what the value of the
  257. first pointer might be. This lack of knowledge prevents the compiler
  258. from carrying out optimizations that otherwise might destroy the ordering
  259. guarantees that RCU depends on. And the volatile cast in rcu_dereference()
  260. should prevent the compiler from guessing the value.
  261. But without rcu_dereference(), the compiler knows more than you might
  262. expect. Consider the following code fragment:
  263. struct foo {
  264. int a;
  265. int b;
  266. };
  267. static struct foo variable1;
  268. static struct foo variable2;
  269. static struct foo *gp = &variable1;
  270. void updater(void)
  271. {
  272. initialize_foo(&variable2);
  273. rcu_assign_pointer(gp, &variable2);
  274. /*
  275. * The above is the only store to gp in this translation unit,
  276. * and the address of gp is not exported in any way.
  277. */
  278. }
  279. int reader(void)
  280. {
  281. struct foo *p;
  282. p = gp;
  283. barrier();
  284. if (p == &variable1)
  285. return p->a; /* Must be variable1.a. */
  286. else
  287. return p->b; /* Must be variable2.b. */
  288. }
  289. Because the compiler can see all stores to "gp", it knows that the only
  290. possible values of "gp" are "variable1" on the one hand and "variable2"
  291. on the other. The comparison in reader() therefore tells the compiler
  292. the exact value of "p" even in the not-equals case. This allows the
  293. compiler to make the return values independent of the load from "gp",
  294. in turn destroying the ordering between this load and the loads of the
  295. return values. This can result in "p->b" returning pre-initialization
  296. garbage values.
  297. In short, rcu_dereference() is -not- optional when you are going to
  298. dereference the resulting pointer.