Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add method field to CDXJ #107

Closed
wants to merge 1 commit into from
Closed

Add method field to CDXJ #107

wants to merge 1 commit into from

Conversation

edsu
Copy link

@edsu edsu commented May 17, 2022

Allow for a method field in the CDXJ, which can be included in pywb's CDXJ when a WARC record was created with a POST.

Fixes #106

Allow for a a `method` property in the CDXJ, which is occasionally
emitted by pywb.

Fixes nla#106
@edsu
Copy link
Author

edsu commented May 17, 2022

I guess this just parses it, but doesn't serialize the data to the db. Changing this to a draft for now.

@edsu edsu marked this pull request as draft May 17, 2022 12:07
@edsu
Copy link
Author

edsu commented May 17, 2022

@ato would adding method to the db require a new indexVersion (5)?

@ato
Copy link
Member

ato commented May 17, 2022

Yeah. New fields requires a new version number so that the reading code knows what to expect. The approach I was thinking about was to add an extra serialized field that stores a blob of json with all the extra fields (since as far as I know they could be arbitrarily complex json structures).

@steph-nb
Copy link

steph-nb commented Dec 1, 2022

you also have to add a requestBody field, no?

java.lang.IllegalArgumentException: no such capture field: requestBody
at outbackcdx.Capture.put(Capture.java:554)
at outbackcdx.Capture.fromCdxjLine(Capture.java:437)
at outbackcdx.Capture.fromCdxLine(Capture.java:388)
at outbackcdx.Webapp.post(Webapp.java:249)
at outbackcdx.Webapp.lambda$new$3(Webapp.java:102)
at outbackcdx.Web$Route.handle(Web.java:312)
at outbackcdx.Web$Router.handle(Web.java:236)
at outbackcdx.Webapp.handle(Webapp.java:594)
at outbackcdx.Web$Server.serve(Web.java:50)
at outbackcdx.NanoHTTPD$HTTPSession.execute(NanoHTTPD.java:848)
at outbackcdx.NanoHTTPD$1$1.run(NanoHTTPD.java:207)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

@ato
Copy link
Member

ato commented Jun 12, 2023

I'm going to close this since we now support arbitrary CDXJ fields (commit 9d73df3) which I think should cover this. Feel free to reopen if I'm wrong though!

@ato ato closed this Jun 12, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CDXJ: Error: no such capture field: method
3 participants