Skip to content

Dev #29

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merged
merged 128 commits into from
Jun 4, 2024
Merged

Dev #29

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
128 commits
Select commit Hold shift + click to select a range
7928088
add deploy workflow
lachlanglen Apr 29, 2024
c7569ec
fix workflows location
lachlanglen Apr 29, 2024
1087fd5
fix ec2 directory path
lachlanglen Apr 29, 2024
4bd0940
add workflow debug logs
lachlanglen Apr 29, 2024
648e5b8
change env. to vars.
lachlanglen Apr 29, 2024
5f6c725
change all env. to vars.
lachlanglen Apr 29, 2024
152cb35
add appspec
lachlanglen Apr 29, 2024
f740c64
update workflow to use codedeploy
lachlanglen Apr 30, 2024
757f774
remove whitespace in deploy workflow
lachlanglen Apr 30, 2024
5eff58c
update after_install script
lachlanglen Apr 30, 2024
160af1a
set correct file permissions in after_install.sh
lachlanglen Apr 30, 2024
9661137
add timetstamp to after_install logs
lachlanglen Apr 30, 2024
a811354
add log to after_install
lachlanglen Apr 30, 2024
88a8cd3
fix permissions settings in after_install
lachlanglen Apr 30, 2024
4715bc6
add elastic IP to allowed hosts
lachlanglen Apr 30, 2024
948a52e
remove permissions logging from after_install
lachlanglen Apr 30, 2024
42a976f
update nginx group permissions
lachlanglen Apr 30, 2024
fea902f
add poetry install to after_install.sh
lachlanglen Apr 30, 2024
3d2fbc0
remove old code from after_install
lachlanglen Apr 30, 2024
5f06c5d
update donation admin to fix n+1 query issue
lachlanglen Apr 30, 2024
e2ad39c
use tz aware datetime
lachlanglen Apr 30, 2024
b9dddc5
specify start block for debugging
lachlanglen Apr 30, 2024
1221100
trigger redeployment
lachlanglen May 1, 2024
8aeb330
fix settings.py for local celery
lachlanglen May 1, 2024
1c69d33
refactor batch function, edit model
Prometheo May 2, 2024
a04c04f
correct start block
Prometheo May 2, 2024
60801ac
feat: make donation entries update_or_create
Prometheo May 3, 2024
d03c11a
update start block for debugging
lachlanglen May 3, 2024
a615839
Merge pull request #20 from PotLock/refactor-batch-dpnation
lachlanglen May 3, 2024
5f66bc4
debug logs for after_install migration check
lachlanglen May 3, 2024
7024a41
migration check debugging
lachlanglen May 3, 2024
002d94b
load env vars in after_install
lachlanglen May 3, 2024
6d4e576
remove old code
lachlanglen May 3, 2024
b05e457
display donation id on admin
lachlanglen May 3, 2024
544e2c0
fix donation id on admin dashboard
lachlanglen May 3, 2024
6337caa
fix typo in after_install
lachlanglen May 3, 2024
5647bb2
fix TokenHistoricalPrice price_usd null issue
lachlanglen May 3, 2024
9550888
add more debug logs
lachlanglen May 3, 2024
567292b
change debug to info
lachlanglen May 3, 2024
dce6e72
fix typo
lachlanglen May 3, 2024
680d5a0
use logger.debug
lachlanglen May 3, 2024
6ce7d63
revert to logger.info
lachlanglen May 3, 2024
e13d469
fix coingecko api error handling
lachlanglen May 3, 2024
34b86d5
add error test
lachlanglen May 3, 2024
e54861a
remove ERROR test
lachlanglen May 3, 2024
c8583c1
save correct timestamp on TokenHistoricalPrice record
lachlanglen May 3, 2024
03559c7
get start block from cache
lachlanglen May 3, 2024
a226c70
fix None to decimal conversion bug
lachlanglen May 6, 2024
a75da81
add action_result & type unique_together constraint on Activity
lachlanglen May 6, 2024
da4ca37
create or update activity even if donation already exists
lachlanglen May 6, 2024
077090d
await aupdate_or_create Activity call
lachlanglen May 6, 2024
c4afcf1
remove manual start block setting
lachlanglen May 6, 2024
739df78
restart indexer at 105_159_364
lachlanglen May 6, 2024
9ecc002
restart indexer at 105,363,093
lachlanglen May 6, 2024
1fb42f7
restart indexer at 105534694
lachlanglen May 6, 2024
0a2de4b
restart indexer from current block height
lachlanglen May 6, 2024
649e9dc
add celery recurring task for account totals
lachlanglen May 6, 2024
51cb828
update after_install to include celery beat service
lachlanglen May 6, 2024
f6883e4
use logger in update_account_statistics
lachlanglen May 6, 2024
39dd68d
add jobs logger
lachlanglen May 6, 2024
04a4619
debugging account stats task
lachlanglen May 6, 2024
02f6157
fix typo in account stats job
lachlanglen May 6, 2024
b1d07c4
remove transaction.atomic in account stats task
lachlanglen May 6, 2024
d803716
update after_install and celery scripts to handle beat start/stop
lachlanglen May 7, 2024
0583635
remove excess logs for update_account_statistics job
lachlanglen May 7, 2024
c5df889
add celery-beat to restart in after_install
lachlanglen May 7, 2024
2c243f6
remove celery-beat from after_install
lachlanglen May 7, 2024
048ea99
move fetch_usd_prices to Donation model method
lachlanglen May 7, 2024
de07644
add fetch_usd_prices to celery beat schedule
lachlanglen May 7, 2024
96c4afe
add celery-beat stop/restart to after_install
lachlanglen May 7, 2024
d1180aa
make fetch_usd_prices task synchronous
lachlanglen May 7, 2024
92c1915
use sync db methods in Donation model methods
lachlanglen May 7, 2024
d0f384d
allow None chef_fee on Donation
lachlanglen May 7, 2024
1e52096
add migration to set chef_fee to None for existing donations
lachlanglen May 7, 2024
58d560c
fix int None issue in fetch_usd_prices
lachlanglen May 7, 2024
12dc03b
return decimal from Token.format_price
lachlanglen May 7, 2024
3ca9a75
remove redundant log from fetch_usd_prices
lachlanglen May 7, 2024
2d0ddaf
fix decimal * flot issue & restart indexer from block 105_854_538
lachlanglen May 8, 2024
27ac54c
update usd prices job to check for no protocol_fee_usd value
lachlanglen May 8, 2024
d296674
remove manual start block setting
lachlanglen May 8, 2024
4f1ce84
fix donations query in fetch_usd_prices
lachlanglen May 8, 2024
e9eb35a
Backfill compatible (#23)
Prometheo May 9, 2024
a7de2d4
restart at block 106388588
lachlanglen May 10, 2024
1959ca0
restart at block 106388588
lachlanglen May 10, 2024
d653498
restart indexer at 106431915
lachlanglen May 10, 2024
aa6a467
restart at 106435734
lachlanglen May 10, 2024
875bdb7
add pattern matching for pot subaccount
lachlanglen May 13, 2024
fd0e2d2
add cors headers & localhost support
lachlanglen May 15, 2024
137e348
fix virtual env usage in after_install.sh
lachlanglen May 15, 2024
466c093
update after_install.sh
lachlanglen May 15, 2024
7d83b98
revert after_install
lachlanglen May 20, 2024
7b8f79a
update start block to 110046700
lachlanglen May 20, 2024
b8f87aa
avoid logging WorkerLostErrors to Sentry
lachlanglen May 20, 2024
fa5c4bc
update start block to 111941200
lachlanglen May 20, 2024
e83dda8
update start block to 112034320
lachlanglen May 20, 2024
f61cc0d
update start block to 112158850
lachlanglen May 20, 2024
eaf0cf3
update start block to 112461673
lachlanglen May 20, 2024
c2cc367
fix display of admins & wl deployers in PotFactory detail (admin)
lachlanglen May 20, 2024
6456274
update verbose name plurals
lachlanglen May 20, 2024
eeeb00e
add Pot custom form
lachlanglen May 20, 2024
898a512
add admin for PotApplicationReview
lachlanglen May 20, 2024
16e0468
comment out worker_shutdown_handler
lachlanglen May 20, 2024
fb4614f
fix tuple error when creating pot, update start block to 112961757 to…
lachlanglen May 21, 2024
e160c50
update start block to 112920268
lachlanglen May 22, 2024
cb8ed9b
update start block to 113030298
lachlanglen May 22, 2024
3e4678b
update start block to 113042335
lachlanglen May 22, 2024
fcd90b8
update start block to 113058915
lachlanglen May 22, 2024
12e0db4
update start block to 113066329
lachlanglen May 22, 2024
6307a89
update start block to 113155075
lachlanglen May 22, 2024
19e1916
update start block to 113284717
lachlanglen May 22, 2024
93db4c0
update start block to 113455493
lachlanglen May 22, 2024
ee1827d
update start block to 113613466
lachlanglen May 22, 2024
ff92358
update start block to 113614770
lachlanglen May 22, 2024
fc024b9
update start block to 114027103
lachlanglen May 22, 2024
a7f097a
update start block to 114037338
lachlanglen May 22, 2024
c827597
update start block to 114056625
lachlanglen May 22, 2024
ba8c3e9
handle None pot application & update start block
lachlanglen May 23, 2024
48b20ec
use coingecko pro api
lachlanglen May 23, 2024
c7274aa
choose LakeConfig network based on PL_ENVIRONMENT
lachlanglen May 29, 2024
a8fd173
Testnet -> Dev (#26)
lachlanglen May 31, 2024
a95d83e
add debug logs
lachlanglen May 31, 2024
cd2f0b2
add logs
lachlanglen May 31, 2024
7d770ab
fix appspec destination for dev
lachlanglen May 31, 2024
1d81df1
clean up destination directory before installing in deployment (only …
lachlanglen May 31, 2024
41217ed
add clean_destination scripts
lachlanglen May 31, 2024
0bf1a26
remove BeforeInstall clean destination step
lachlanglen Jun 1, 2024
224125b
Testnet (#27)
lachlanglen Jun 4, 2024
2e159f9
Testnet (#28)
lachlanglen Jun 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions .github/workflows/deploy-dev.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: Dev deploy to EC2 on Push

on:
push:
branches: [dev]

env:
AWS_REGION: "us-east-1"

# Permission can be added at job level or workflow level
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout
jobs:
DeployToCodeDeploy:
runs-on: ubuntu-latest
steps:
- name: Git clone the repository
uses: actions/checkout@v3

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1.7.0
with:
role-to-assume: arn:aws:iam::471112976510:role/GitHubAction-AssumeRoleWithAction
role-session-name: GitHub_to_AWS_via_FederatedOIDC
aws-region: ${{ env.AWS_REGION }}

- name: Generate appspec.yml for dev
run: cp appspec-dev.yml appspec.yml

- name: Set environment variables
id: vars
run: |
echo "DATETIME=$(date +'%Y-%m-%d_%H-%M-%S')" >> $GITHUB_ENV
echo "FILENAME=django-indexer-dev-${DATETIME}.zip" >> $GITHUB_ENV
echo "S3_BUCKET=django-indexer-dev" >> $GITHUB_ENV

- name: Create zip of repository
run: zip -r "${{ env.FILENAME }}" .

- name: Upload repository to S3
run: aws s3 cp "${{ env.FILENAME }}" "s3://${{ env.S3_BUCKET }}/"

- name: Create CodeDeploy Deployment
id: deploy
run: |
aws deploy create-deployment \
--application-name django-indexer \
--deployment-group-name django-indexer-dev-group \
--deployment-config-name CodeDeployDefault.AllAtOnce \
--s3-location bucket=${{ env.S3_BUCKET }},bundleType=zip,key=${{ env.FILENAME }}
66 changes: 66 additions & 0 deletions .github/workflows/deploy-testnet.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
name: Testnet deploy to EC2 on Push

on:
push:
branches: [testnet]

env:
AWS_REGION: "us-east-1"

# Permission can be added at job level or workflow level
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout
jobs:
DeployToCodeDeploy:
runs-on: ubuntu-latest
steps:
- name: Git clone the repository
uses: actions/checkout@v3

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1.7.0
with:
role-to-assume: arn:aws:iam::471112976510:role/GitHubAction-AssumeRoleWithAction
role-session-name: GitHub_to_AWS_via_FederatedOIDC
aws-region: ${{ env.AWS_REGION }}

- name: Generate appspec.yml for testnet
run: cp appspec-testnet.yml appspec.yml

- name: Set environment variables
id: vars
run: |
echo "DATETIME=$(date +'%Y-%m-%d_%H-%M-%S')" >> $GITHUB_ENV
echo "FILENAME=django-indexer-testnet-${DATETIME}.zip" >> $GITHUB_ENV
echo "S3_BUCKET=django-indexer-testnet" >> $GITHUB_ENV

- name: Create zip of repository
run: zip -r "${{ env.FILENAME }}" .

- name: Upload repository to S3
run: aws s3 cp "${{ env.FILENAME }}" "s3://${{ env.S3_BUCKET }}/"

- name: Create CodeDeploy Deployment
id: deploy
run: |
aws deploy create-deployment \
--application-name django-indexer-testnet \
--deployment-group-name django-indexer-testnet-group \
--deployment-config-name CodeDeployDefault.AllAtOnce \
--s3-location bucket=${{ env.S3_BUCKET }},bundleType=zip,key=${{ env.FILENAME }}

# - name: Create zip of repository
# run: zip -r django-indexer-testnet.zip .

# - name: Upload repository to S3
# run: aws s3 cp django-indexer-testnet.zip s3://django-indexer-testnet/

# - name: Create CodeDeploy Deployment
# id: deploy
# run: |
# aws deploy create-deployment \
# --application-name django-indexer-testnet \
# --deployment-group-name django-indexer-testnet-group \
# --deployment-config-name CodeDeployDefault.AllAtOnce \
# --s3-location bucket=django-indexer-testnet,bundleType=zip,key=django-indexer-testnet.zip
14 changes: 8 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
- [Steps to run:](#steps-to-run)
- [Env vars example](#env-vars-example)
- [API Basics](#api-basics)
- [Base URL](#base-url)
- [Authorization](#authorization)
- [Error Responses](#error-responses)
- [Pagination](#pagination)
- [Base URL](#base-url)
- [Authorization](#authorization)
- [Error Responses](#error-responses)
- [Pagination](#pagination)
- [API Endpoints](#api-endpoints)
- [`Account` endpoints](#account-endpoints)
- [✅ Get all accounts: `GET /accounts` (paginated)](#-get-all-accounts-get-accounts-paginated)
Expand Down Expand Up @@ -52,6 +52,7 @@
- If for some reason this doesn't kill any active celery tasks, run `ps auxww | grep 'celery' | grep -v grep` and kill resulting PIDs

Extra commands that might come in useful:

- Purge celery queue (`celery -A base purge`)

### Env vars example
Expand All @@ -77,13 +78,14 @@ export PL_SENTRY_DSN=

#### Base URL

`/api/v1/`
**dev (mainnet):** `https://dev.potlock.io/api/v1/`
**testnet:** `https://test-dev.potlock.io/api/v1/`

#### Authorization

This is a public, read-only API and as such does not currently implement authentication or authorization.

Rate limits of (FILL THIS IN) are enforced to ensure service for all users.
Rate limits of 100 requests/min are enforced to ensure service for all users.

#### Error Responses

Expand Down
47 changes: 37 additions & 10 deletions accounts/admin.py
Original file line number Diff line number Diff line change
@@ -1,25 +1,52 @@
from django.contrib import admin

from .models import Account


@admin.register(Account)
class AccountAdmin(admin.ModelAdmin):
list_display = ('id', 'total_donations_in_usd', 'total_donations_out_usd', 'total_matching_pool_allocations_usd', 'donors_count')
search_fields = ('id',) # Allow searching by account address
list_filter = ('total_donations_in_usd', 'total_donations_out_usd') # Filter by donation amounts
ordering = ('-total_donations_in_usd',) # Default ordering
list_display = (
"id",
"total_donations_in_usd",
"total_donations_out_usd",
"total_matching_pool_allocations_usd",
"donors_count",
)
search_fields = ("id",) # Allow searching by account address
list_filter = (
"total_donations_in_usd",
"total_donations_out_usd",
) # Filter by donation amounts
ordering = ("-total_donations_in_usd",) # Default ordering

# Optionally, format decimal fields for better readability in the admin
def total_donations_in_usd_display(self, obj):
return "${:,.2f}".format(obj.total_donations_in_usd)
total_donations_in_usd_display.admin_order_field = 'total_donations_in_usd'
total_donations_in_usd_display.short_description = 'Total Donations Received (USD)'

total_donations_in_usd_display.admin_order_field = "total_donations_in_usd"
total_donations_in_usd_display.short_description = "Total Donations Received (USD)"

def total_donations_out_usd_display(self, obj):
return "${:,.2f}".format(obj.total_donations_out_usd)
total_donations_out_usd_display.admin_order_field = 'total_donations_out_usd'
total_donations_out_usd_display.short_description = 'Total Donations Sent (USD)'

total_donations_out_usd_display.admin_order_field = "total_donations_out_usd"
total_donations_out_usd_display.short_description = "Total Donations Sent (USD)"

def total_matching_pool_allocations_usd_display(self, obj):
return "${:,.2f}".format(obj.total_matching_pool_allocations_usd)
total_matching_pool_allocations_usd_display.admin_order_field = 'total_matching_pool_allocations_usd'
total_matching_pool_allocations_usd_display.short_description = 'Total Matching Pool Allocations (USD)'

total_matching_pool_allocations_usd_display.admin_order_field = (
"total_matching_pool_allocations_usd"
)
total_matching_pool_allocations_usd_display.short_description = (
"Total Matching Pool Allocations (USD)"
)

def has_add_permission(self, request):
return False

def has_change_permission(self, request, obj=None):
return False

def has_delete_permission(self, request, obj=None):
return False
44 changes: 33 additions & 11 deletions activities/admin.py
Original file line number Diff line number Diff line change
@@ -1,30 +1,52 @@
from django.contrib import admin
from django.utils.html import format_html
from .models import Activity, Account

from .models import Account, Activity


@admin.register(Activity)
class ActivityAdmin(admin.ModelAdmin):
list_display = ('id', 'signer_address', 'receiver_address', 'timestamp', 'type', 'transaction_link', 'action_result')
list_filter = ('timestamp', 'type', 'signer', 'receiver')
search_fields = ('signer__id', 'receiver__id', 'tx_hash')
date_hierarchy = 'timestamp'
ordering = ('-timestamp',)
list_display = (
"id",
"signer_address",
"receiver_address",
"timestamp",
"type",
"transaction_link",
"action_result",
)
list_filter = ("timestamp", "type", "signer", "receiver")
search_fields = ("signer__id", "receiver__id", "tx_hash")
date_hierarchy = "timestamp"
ordering = ("-timestamp",)

def signer_address(self, obj):
return obj.signer.id
signer_address.admin_order_field = 'signer'
signer_address.short_description = 'Signer Address'

signer_address.admin_order_field = "signer"
signer_address.short_description = "Signer Address"

def receiver_address(self, obj):
return obj.receiver.id
receiver_address.admin_order_field = 'receiver'
receiver_address.short_description = 'Receiver Address'

receiver_address.admin_order_field = "receiver"
receiver_address.short_description = "Receiver Address"

def transaction_link(self, obj):
url = f"https://nearblocks.io?query={obj.tx_hash}"
return format_html('<a href="{}" target="_blank">{}</a>', url, obj.tx_hash)
transaction_link.short_description = 'Transaction Hash' # Sets the column header

transaction_link.short_description = "Transaction Hash" # Sets the column header

# def action_result_summary(self, obj):
# return "Has Result" if obj.action_result else "No Result"
# action_result_summary.short_description = 'Action Result'

def has_add_permission(self, request):
return False

def has_change_permission(self, request, obj=None):
return False

def has_delete_permission(self, request, obj=None):
return False
17 changes: 17 additions & 0 deletions activities/migrations/0002_alter_activity_options.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Generated by Django 5.0.4 on 2024-05-03 10:29

from django.db import migrations


class Migration(migrations.Migration):

dependencies = [
("activities", "0001_initial"),
]

operations = [
migrations.AlterModelOptions(
name="activity",
options={"verbose_name_plural": "Activities"},
),
]
17 changes: 17 additions & 0 deletions activities/migrations/0003_alter_activity_unique_together.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Generated by Django 5.0.4 on 2024-05-06 13:39

from django.db import migrations


class Migration(migrations.Migration):

dependencies = [
("activities", "0002_alter_activity_options"),
]

operations = [
migrations.AlterUniqueTogether(
name="activity",
unique_together={("action_result", "type")},
),
]
2 changes: 2 additions & 0 deletions activities/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,5 @@ class Activity(models.Model):

class Meta:
verbose_name_plural = "Activities"

unique_together = (("action_result", "type"),)
19 changes: 19 additions & 0 deletions appspec-dev.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
version: 0.0
os: linux
files:
- source: /
destination: /home/ec2-user/django-indexer-dev
hooks:
# BeforeInstall:
# - location: scripts/clean_destination_dev.sh
# timeout: 300
# runas: ec2-user
AfterInstall:
- location: scripts/after_install_dev.sh
timeout: 300
runas: ec2-user
# ApplicationStart:
# - location: scripts/application_start.sh
# timeout: 300
# runas: root
# # ValidateService:
20 changes: 20 additions & 0 deletions appspec-testnet.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
version: 0.0
os: linux
files:
- source: /
destination: /home/ec2-user/django-indexer-testnet
hooks:
# # Install:
# BeforeInstall:
# - location: scripts/clean_destination_testnet.sh
# timeout: 300
# runas: ec2-user
AfterInstall:
- location: scripts/after_install_testnet.sh
timeout: 300
runas: ec2-user
# ApplicationStart:
# - location: scripts/application_start.sh
# timeout: 300
# runas: root
# # ValidateService:
16 changes: 16 additions & 0 deletions appspec.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
version: 0.0
os: linux
files:
- source: /
destination: /home/ec2-user/django-indexer
hooks:
# # Install:
AfterInstall:
- location: scripts/after_install.sh
timeout: 300
runas: ec2-user
# ApplicationStart:
# - location: scripts/application_start.sh
# timeout: 300
# runas: root
# # ValidateService:
Loading
Loading