[Audit][SPARK-39749][SQL] Always use plain string representation on casting Decimal to String

[amahussein]

Is your feature request related to a problem? Please describe.
Currently in Spark, casting decimal as string type will result in Strings with exponential notations if the adjusted exponent is less than -6. This is consistent with BigDecimal.toString https://docs.oracle.com/javase/8/docs/api/java/math/BigDecimal.html#toString

After this issue got merged, the casting always uses plain string representation.
To restore the legacy behavior, which uses scientific notation if the adjusted exponent is less than -6, set spark.sql.legacy.castDecimalToString.enabled to true.

Describe the solution you'd like
We need to revisit the cast tests, documentation, and the description of the RAPIDS configuration.

Additional context
apache/spark@c621df269d

EDIT

A followup issue was created for spark-3.4
[SPARK-39749][SQL][FOLLOWUP] Move the new behavior of CAST(DECIMAL AS STRING) under ANSI SQL mode
apache/spark@5bd47640dc