pandas.read_sql_table switches double quotation wrapping keys and values to single quotation in JSON columns

[Magicbeanbuyer]

Code Sample

>>> df = pandas.read_sql_table('table_name', 'postgres:///db_name')
>>> df.head(1)

Problem description

I used pandas.read_sql_table() to fetch a table, in which there is a JSON column. In the returned dataframe, the double quotations wrapping the keys and values are all changed to single quotations.

Example of returned dataframe:

id	name
1	{'first': 'Anna', 'last': 'Smith'}

However, JSON standards requires double quotation, on top of that, other Python package like json doesn't recognize JSON with single quotations.

Expected Output

id	name
1	{"first": "Anna", "last": "Smith"}

Output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.3.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None

pandas           : 0.25.3
numpy            : 1.17.4
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 19.3.1
setuptools       : 42.0.2.post20191203
Cython           : 0.29.14
pytest           : 5.3.1
hypothesis       : None
sphinx           : 2.2.2
blosc            : None
feather          : None
xlsxwriter       : 1.2.6
lxml.etree       : 4.4.2
html5lib         : 1.0.1
pymysql          : None
psycopg2         : 2.7.6.1 (dt dec pq3 ext lo64)
jinja2           : 2.10.3
IPython          : 7.9.0
pandas_datareader: None
bs4              : 4.7.1
bottleneck       : 1.3.1
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.4.2
matplotlib       : 3.1.1
numexpr          : 2.7.0
odfpy            : None
openpyxl         : 3.0.2
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : 0.4.0
scipy            : 1.3.1
sqlalchemy       : 1.3.11
tables           : 3.6.1
xarray           : None
xlrd             : 1.2.0
xlwt             : 1.3.0
xlsxwriter       : 1.2.6

[maxcorbeau]

Same issue here.

pd.show_versions()

INSTALLED VERSIONS

commit : 66e3805
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.13.0-27-generic
Version : #29~20.04.1-Ubuntu SMP Fri Jan 14 00:32:30 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.5
numpy : 1.22.1
pytz : 2021.3
dateutil : 2.8.2
pip : 21.3.1
setuptools : 60.5.0
Cython : 0.29.26
pytest : 6.2.5
hypothesis : None
sphinx : 1.8.5
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.7.1
html5lib : 1.1
pymysql : None
psycopg2 : 2.9.2 (dt dec pq3 ext lo64)
jinja2 : 3.0.3
IPython : 8.0.1
pandas_datareader: None
bs4 : 4.10.0
bottleneck : 1.3.2
fsspec : 2022.01.0
fastparquet : None
gcsfs : None
matplotlib : 3.5.1
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : 6.0.1
pyxlsb : None
s3fs : 2022.01.0
scipy : 1.7.3
sqlalchemy : 1.4.29
tables : 3.6.1
tabulate : 0.8.9
xarray : 0.17.0
xlrd : 2.0.1
xlwt : 1.3.0
numba : None

[datascientist-dribe]

im facing the same issue unfortunately.

[mathieulebasic]

I stumbled upon this issue as well 😢

[brentshulman-silkline]

I solved it, temporarily, by using https://docs.python.org/3/library/ast.html#ast.literal_eval

[loganthomas]

+1