Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Add Chicago Data Sets #235

Closed
2 of 4 tasks
vforgione opened this issue Mar 12, 2018 · 2 comments
Closed
2 of 4 tasks

Add Chicago Data Sets #235

vforgione opened this issue Mar 12, 2018 · 2 comments
Assignees
@vforgione
Labels
Blocked Tasks that are unable to be completed. Task Work that is not bug related.

Comments

@vforgione
Copy link
Member

vforgione commented Mar 12, 2018 •
edited
Loading 

Hi Vince,
 
Sorry for the really late reply but I realized I still owed you something from the last Open Grid / Plenario call – some ideas for test Chicago datasets for Plenario 2. I actually don’t remember if there were any specific criteria but some that seem as if they might be good are:
 
http://plenar.io/explore/event/divvy_trips (fairly big, although changes rarely)
http://plenar.io/explore/event/crimes_2001_to_present
http://plenar.io/explore/event/beach_lab_data_culture_tests, http://plenar.io/explore/event/beach_lab_data_dna_tests (out of season right now and not meant to test anything in particular but since we had some issues with them)
http://plenar.io/explore/event/relocated_vehicles (small but frequent update)
https://data.cityofchicago.org/Transportation/Chicago-Traffic-Tracker-Historical-Congestion-Esti/sxs8-h27x (Let’s not put this on the live Plenario yet since this is just a soft launch. For that matter, maybe you don’t want to use it as a test for Plenario 2 since we are still testing the dataset, itself, and it might change. However, it is big, frequently updated, has multiple location columns, and may present other issues to push your process a bit.)
 
Thanks.
 
Jon
  • divvy trips
  • crime data
  • beach lab - culture
  • beach lab - dna
@vforgione vforgione self-assigned this Mar 12, 2018
@vforgione vforgione added this to the To a One-dot Release milestone Jun 28, 2018
@vforgione vforgione mentioned this issue Jun 28, 2018
Closed
@vforgione
Copy link
Member Author

Divvy and Crimes in particular are problematic. The server needs to render the entire data set before it can start sending it to us. I put an email into Jon over at the city's data office about work arounds for this.

@vforgione vforgione removed this from the To a One-dot Release milestone Jul 19, 2018
@vforgione vforgione added Task Work that is not bug related. Blocked Tasks that are unable to be completed. and removed blocked labels Jul 19, 2018
@vforgione
Copy link
Member Author

Blocked until we start addressing #377

vforgione pushed a commit that referenced this issue Aug 21, 2018
Fixed Meta Detail Rendering Bug
181f7d4 
The original signature didn't account for the function being called when
the data set was still empty. This corrects that.

Updates #235
vforgione pushed a commit that referenced this issue Aug 21, 2018
Set Max Name Length
4704472 
TIL Postgres has a maximum table name length of 63 characters. I set an
upper bound of 58 on the name length of the MetaChangeset. Why 58 and
not 63? Because when we create the materialized view for the data set we
tack on "_view" to the table name.

Updates #235
@vforgione vforgione mentioned this issue Aug 21, 2018
Merged
vforgione pushed a commit that referenced this issue Oct 26, 2018
Modify metas table
b9d4d97 
Adds new columns for Socrata sourced data sets and make the original
source field nullable.

Updates #340
Updates #235
vforgione pushed a commit that referenced this issue Oct 26, 2018
Added socrata fields to meta schema
5c22d47 
Added fields and annotated source url to be default null.

Updates #340
Updates #235
vforgione pushed a commit that referenced this issue Oct 26, 2018
Updated Meta Actions
3a73bef 
Since I totally rewrote the changesets, I needed to update the actions.
And while I was updating how it worked with changesets, I refactored its
methods as well.

This module has smelled for a while and it was time for a change.

Updates #340
Updates #235
vforgione pushed a commit that referenced this issue Oct 26, 2018
Updated internal API calls
7595339 
Many internal MetaActions changed signatures and caused some serious
changes to tests and other internal actions.

Also added back some helper functions.

Updates #340
Updates #235
vforgione pushed a commit that referenced this issue Nov 14, 2018
Toward a sane internal API and Socrata native ingests
ebf36d7 
The internal API was getting really nasty -- we had a bunch of one off
functions that clashed in arity (positional arguments, matches, guards,
options ...).

The web application was also a disaster -- originally I thought it would
make it easier to keep the web, admin and API separate in subapps,
but that ended up making things that much more difficult.

Then that leaves the elephant in the room: Socrata. We've always relied
on them and all of their awful decisions. The changes here in remove
some of the terrible things about Socrata integration and makes
ingesting their data sets a little cleaner.

Breaking Changes:

- total revision of the migrations
- entirely removed the `UserAdminMessage` schema
- entirely removes all the outstanding ETL job stuff
- entirely removes charts -- that was a really stupid idea
- entirely removes exports -- again just a stupid idea
- totally new ingest pipeline
- slimmed down the API (still needs some work0

Closes #235
Closes #340
Closes #360
Closes #361
@vforgione vforgione mentioned this issue Nov 14, 2018
Merged
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@vforgione vforgione

Labels
Blocked Tasks that are unable to be completed. Task Work that is not bug related.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vforgione