Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Support refs -r pinning method #326

Closed
hsanjuan opened this issue Mar 2, 2018 · 1 comment · Fixed by #331
Closed

Support refs -r pinning method #326

hsanjuan opened this issue Mar 2, 2018 · 1 comment · Fixed by #331
Assignees
@hsanjuan
Labels
exp/intermediate Prior experience is likely helpful P2 Medium: Good to have, but can wait until someone steps up

Comments

@hsanjuan
Copy link
Collaborator

hsanjuan commented Mar 2, 2018

It should be possible to configure the ipfs connector to perform pinning with refs -r + pin rather than just
pin. This should allow to perform fetching in parallel.
@hsanjuan hsanjuan added help wanted Seeking public contribution on this issue exp/intermediate Prior experience is likely helpful status/ready Ready to be worked P2 Medium: Good to have, but can wait until someone steps up labels Mar 2, 2018
@hsanjuan
Copy link
Collaborator Author

hsanjuan commented Mar 2, 2018 

<lgierth> Lars hsanjuan: are you importing things with `pin add` or `refs -r`?
18:23:38 <hsanjuan> Hector lgierth: cluster does `pin add`
18:30:04 <lgierth> Lars ah i see, ok
18:30:12 so that's why other pins don't get worked on
18:30:27 or rather, take some time to get worked on
18:48:03 <hsanjuan> Hector lgierth: refs -r will fetch everything and works in parallel ?
18:50:42 <lgierth> Lars yeah. it's fetches slightly differently from pin (more depth-first than breadth-first) but yes you can do a ton of `refs -r` calls in parallel
18:51:19 why keeps insisting that pin fetches faster, but i'll claim that refs fetches more consistently fast

@hsanjuan hsanjuan self-assigned this Mar 6, 2018
@hsanjuan hsanjuan added status/in-progress In progress and removed help wanted Seeking public contribution on this issue status/ready Ready to be worked labels Mar 6, 2018
hsanjuan added a commit that referenced this issue Mar 7, 2018
@hsanjuan
Feat #326: Adds "refs -r" pinning method support + multiple pin workers
8f944d3 
This fixes #326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.

It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as

Documentation for the configuration file has been updated.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
hsanjuan added a commit that referenced this issue Mar 7, 2018
@hsanjuan
Feat #326: Adds "refs -r" pinning method support + multiple pin workers
e359602 
This fixes #326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.

It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as

Documentation for the configuration file has been updated.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
hsanjuan added a commit that referenced this issue Mar 7, 2018
@hsanjuan
Feat #326: Adds "refs -r" pinning method support + multiple pin workers
4ff53ab 
This fixes #326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.

It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as #308
is still pending.

Documentation for the configuration file has been updated.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
hsanjuan added a commit that referenced this issue Mar 7, 2018
@hsanjuan
Feat #326: Adds "refs -r" pinning method support + multiple pin workers
6ce5baf 
This fixes #326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.

It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as #308
is still pending.

Documentation for the configuration file has been updated.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
@hsanjuan hsanjuan mentioned this issue Mar 7, 2018
Merged
hsanjuan added a commit that referenced this issue Mar 9, 2018
@hsanjuan
Feat #326: Adds "refs -r" pinning method support + multiple pin workers
fb4812e 
This fixes #326. It adds a new `pin_method` configuration option to the
`ipfshttp` component allows to configure it to perform `refs -r <cid>` before
the `pin/add` call. By fetching content before pinning, we don't have
a global lock in place, and we can have several pin-requests to
ipfs in parallel.

It also adds a `concurrent_pins` option to the pin tracker, which
launches more pin workers so it can potentially trigger more pins at
the same time. This is a minimal intervention in the pintracker as #308
is still pending.

Documentation for the configuration file has been updated.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>
@hsanjuan hsanjuan added need/review Needs a review and removed status/in-progress In progress labels Mar 9, 2018
hsanjuan added a commit that referenced this issue Mar 10, 2018
@hsanjuan
Merge pull request #331 from ipfs/feat/pin-refs
c276c31 
Feat #326: Adds "refs -r" pinning method support + multiple pin workers
@ghost ghost removed the need/review Needs a review label Mar 10, 2018
@hsanjuan hsanjuan mentioned this issue Mar 12, 2018
Closed
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@hsanjuan hsanjuan

Labels
exp/intermediate Prior experience is likely helpful P2 Medium: Good to have, but can wait until someone steps up
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feat #326: Adds "refs -r" pinning method support + multiple pin workers ipfs-cluster/ipfs-cluster
1 participant
@hsanjuan