-
Notifications
You must be signed in to change notification settings - Fork 70
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
garbage collection seems to be broken #198
Comments
Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
Finally, remember to use https://discuss.ipfs.io if you just need general support. |
It looks like the query logic isn't properly parsing keys. The query logic is known to have a bunch of issues, fixed in #39. However, that implements naive query logic (in some edge cases) which make the test-cases take way too long. |
The |
I've successfully merged release v0.8.0 into the #39 and made adjustments to certain code logic to ensure the query functions correctly. The changes made are as follows:
Here's the update I applied: diff --git a/s3.go b/s3.go
index 6205220..02f6269 100644
--- a/s3.go
+++ b/s3.go
@@ -215,6 +215,18 @@ func (s *S3Bucket) Query(ctx context.Context, q dsq.Query) (dsq.Results, error)
// Normalize the path and strip the leading / as S3 stores values
// without the leading /.
prefix := ds.NewKey(q.Prefix).String()[1:]
+ prefix = s.s3Path(prefix)
+ if strings.HasPrefix(prefix, "/") {
+ prefix = prefix[1:]
+ }
+ dirPrefix := s.s3Path("") + "/"
+ if strings.HasPrefix(dirPrefix, "/") {
+ dirPrefix = dirPrefix[1:]
+ }
+ dirPrefixLen := len(dirPrefix)
+ if len(prefix) < dirPrefixLen {
+ prefix = dirPrefix
+ }
sent := 0
queryLimit := func() int64 {
@@ -226,7 +238,7 @@ func (s *S3Bucket) Query(ctx context.Context, q dsq.Query) (dsq.Results, error)
resp, err := s.S3.ListObjectsV2WithContext(ctx, &s3.ListObjectsV2Input{
Bucket: aws.String(s.Bucket),
- Prefix: aws.String(s.s3Path(prefix)),
+ Prefix: aws.String(prefix),
Delimiter: aws.String("/"),
MaxKeys: aws.Int64(queryLimit()),
})
@@ -250,7 +262,7 @@ func (s *S3Bucket) Query(ctx context.Context, q dsq.Query) (dsq.Results, error)
resp, err = s.S3.ListObjectsV2WithContext(ctx, &s3.ListObjectsV2Input{
Bucket: aws.String(s.Bucket),
- Prefix: aws.String(s.s3Path(prefix)),
+ Prefix: aws.String(prefix),
Delimiter: aws.String("/"),
MaxKeys: aws.Int64(queryLimit()),
ContinuationToken: resp.NextContinuationToken,
@@ -260,8 +272,9 @@ func (s *S3Bucket) Query(ctx context.Context, q dsq.Query) (dsq.Results, error)
}
}
+ key := (*resp.Contents[index].Key)[dirPrefixLen:]
entry := dsq.Entry{
- Key: ds.NewKey(*resp.Contents[index].Key).String(),
+ Key: ds.NewKey(key).String(),
Size: int(*resp.Contents[index].Size),
}
if !q.KeysOnly { This change has enabled both |
Garbage collection appears to be broken with this plugin.
I have a working implementation of things where I can add / pin content, list content, get the size of the repo etc.
However running

ipfs repo gc
returns nothing and shows this in the daemon logs:No content is deleted from the s3 bucket when this happens.
Oddly I can run
ipfs block rm CID
just fine and the content is removed from the s3 bucketThe text was updated successfully, but these errors were encountered: