A quick way to get started with Australian Bureau of Statistics (ABS) Census 2021 data and boundaries. Note: Boundaries are available in both datums (GDA94 and GDA2020)
- Run the load-census Python script and build the database schemas in a single step on a Mac or Linux machine
- Pull the database from Docker Hub and run it in a container
- Download the Postgres dump files and restore them in your database.
Running the Python script takes 15-30 minutes on a Postgres server (or desktop) configured for performance.
To get a good load time you'll need to configure your Postgres server for performance. There's a good guide here, noting its old and the memory parameters can be beefed up if you have the RAM.
- Postgres 14+ with PostGIS 3.2+ (tested on 14.10 on macOS Sonoma)
- Add the Postgres bin directory to your system PATH
- Python 3.6+ with Psycopg 3 & Pandas packages installed
- Run the
xx_run_all.sh
script in thesupporting-files/processing
folder. It will download the data and boundary files from the ABS and import them into Postgres in a single step. - Come back in 15-30 minutes and enjoy!
Alternately - you can download the files manually yourself, import the boundary Shapefiles or GeoPackages using GDAL and then import the data using the main script load-census.py
.
The behaviour of the main census-loader script can be controlled by specifying various command line options to the script. Supported arguments are:
--census-data-path
specifies the path to the extracted Census metadata and data tables (eg *.xlsx and *.csv files). This directory must be accessible by the Postgres server, and the corresponding local path for the server to this directory may need to be set via thelocal-server-dir
argument
--pghost
the host name for the Postgres server. This defaults to thePGHOST
environment variable if set, otherwise defaults tolocalhost
.--pgport
the port number for the Postgres server. This defaults to thePGPORT
environment variable if set, otherwise5432
.--pgdb
the database name for Postgres server. This defaults to thePGDATABASE
environment variable if set, otherwisegeo
.--pguser
the username for accessing the Postgres server. This defaults to thePGUSER
environment variable if set, otherwisepostgres
.--pgpassword
password for accessing the Postgres server. This defaults to thePGPASSWORD
environment variable if set, otherwisepassword
.
--data-schema
schema name to store Census data tables in. Defaults tocensus_2021_data
. You will need to change this argument if you set--census-year=2011
--max-processes
specifies the maximum number of parallel processes to use for the data load. Set this to the number of cores on the Postgres server minus 2, but limit to 12 if 16+ cores - there is minimal benefit beyond 12. Defaults to 3.
python load-census.py --census-data-path="C:\temp\census_2021_data"
Loads the 2021 Census data using a maximum of 4 parallel processes (the default) into the default schema. Census data archives have been extracted to the folder C:\temp\census_2021_data
.
python load-census.py --max-processes=8 --data-schema=census_2021_stats --census-data-path="C:\temp\census_2011_data"
Loads the 2021 Census data using a maximum of 8 parallel processes into a renamed schema. Census data archives have been extracted to the folder C:\temp\census_2011_data
.
When using the resulting data from this process - you will need to adhere to the ABS data attribution requirements for the Census and ASGS data, as per the Creative Commons (Attribution) license.
- The scripts will DROP ALL TABLES and recreate them using CASCADE; meaning you'll LOSE YOUR VIEWS if you have created any! If you want to keep the existing data - you'll need to change the schema names in the script or use a different database
- Whilst you can choose which schema to load the data into, I haven't QA'd the permutations. Stick with the defaults if you have limited Postgres experience
Download and run a Docker container with Census data and ASGS boundaries ready to go in a Postgres database.
- In your docker environment pull the image using
docker pull minus34/censusloader:latest
- Run using
docker run --publish=5433:5432 minus34/censusloader:latest
- Access Postgres in the container via port
5433
. Default login is - user:postgres
, password:password
Note: the compressed Docker image is 8Gb, uncompressed is 27Gb
WARNING: The default postgres superuser password is insecure and should be changed using:
ALTER USER postgres PASSWORD '<something a lot more secure>'
Download Postgres dump files and restore them in your database.
Should take 15-30 minutes.
- Postgres 14+ with PostGIS 3.2+
- A knowledge of Postgres pg_restore parameters
- Download census_2021_data.dmp (~1.1Gb)
- Download census_2021_bdys_gda94.dmp or census_2021_bdys_gda2020.dmp (~1.5Gb)
- Edit the
restore-census-schemas.bat
or 'restore-census-schemas.sh' script in the 'supporting-files' folder for your database parameters and for the location of pg_restore - Run the script, come back in 15-30 minutes and enjoy!
Source: Australian Bureau of Statistics