WiP: add csv parsing and start config support #4
Conversation
Fix bug in argument parser
Data getter, syncer and plotter.
|
@mlesniew Take a look. This is a semi-finished state. It works on my system, actually it replaced a previous solution (very "convoluted"). There are also workarounds for some quirks/bugs in the tauron API. I did not check, but the json API may need similar fixes (different end-of-day time for single and multiple days: I use 0-23 convention). The examples may be usable. I suspect you would prefer to merge only some of this, if at all. |
|
Thanks for the submission. Getting the readings using the CSV seems better than using the JSON API. It allows getting both production and consumption readings with one request. The new JSON API doesn't seem to have that option. I would like to include the code to use the CSV API and make that the default, leaving the JSON variant that can be used when it's explicitly requested. While the current code works perfectly, I would like to tidy it up a bit before merging to master. I will try to add comments about possible minor improvements soon, especially in There is however one higher level improvement that I'd like to suggest. With the current implementation we have two methods in Regarding the examples -- I didn't really have time yet to review them carefully, but we can add them to the repo, especially for their educational aspect (so that others can learn how to use the module from them). I would really appreciate if you could submit examples in a separate pull request if you'd like me to merge them. Also, please try to put them in separate directories under You can have a look at an example script that I've added in meantime: https://github.com/mlesniew/elicznik/tree/master/examples/google-sheets, all feedback is welcome. Before the final merge, please make sure to squash commits into logically complete bits and rebase to master, so that we can do a fast forward merge. |
| "form[oze]": 1, | ||
| "form[fileType]": "CSV", # or "XLS" | ||
| }, | ||
| ).content.decode().split('\n') |
There was a problem hiding this comment.
.content.decode() should be replaced by .text. It will decode the bytes to a string automatically and it should use the right encoding based on HTTP response headers.
|
|
||
| def get_data(self, start_date, end_date=None): | ||
| end_date = end_date or start_date | ||
| data = csv.reader(self.get_raw_data(start_date, end_date)[1:], delimiter=';') |
There was a problem hiding this comment.
Use csv.DictReader instead of csv.reader. It will parse the header row automatically and produce a dict for each row.
| continue | ||
| date, hour = t.split() | ||
| h, m = hour.split(':') | ||
| timestamp = datetime.datetime.strptime(date, "%d.%m.%Y") |
There was a problem hiding this comment.
It should be possible to parse the whole time stamp in one go using strptime. However, I'm not sure if it will work correctly with dates like 16.01.2023 24:00.
| timestamp += datetime.timedelta(hours=int(h), minutes=int(m)) | ||
| # Skip records outside a single day block | ||
| if start_date == end_date and timestamp.day != start_date.day: | ||
| print(f'Skip {timestamp} not within {start_date}.') |
There was a problem hiding this comment.
Avoid print statements here. For anything critical, raise an exception. logging could be used for everything else, but the module is probably to simple to introduce that.
|
Thanks. I will. Next week please ;). |
Sync with upstream
This is a start of small tweaks:
This is to keep track and share. I will be working on it.