Skip to content

Add Windows support #108

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Merged
merged 4 commits into from
Dec 23, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .cargo/config
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
[build]
# Informs the linker that some of the symbols for Postgres won't be available
# until runtime on the dynamic library load.
# until runtime on the dynamic library load. Flags differ based on target family

[target.'cfg(unix)']
rustflags = "-C link-arg=-undefineddynamic_lookup"

[target.'cfg(windows)']
rustflags = "-C link-arg=/FORCE"
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,11 @@ First install Postgres. The build should be able to find the directory for the P
For the dynamic library to compile, your project should also have `.cargo/config` file with content:

```toml
[build]
[target.'cfg(unix)']
rustflags = "-C link-arg=-undefineddynamic_lookup"

[target.'cfg(windows)']
rustflags = "-C link-arg=/FORCE"
```

This informs the linker that some of the symbols for Postgres won't be available until runtime on the dynamic library load.
Expand Down
92 changes: 90 additions & 2 deletions pg-extend/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,7 @@ fn main() {
.collect(),
);

let bindings = bindgen::Builder::default()
.clang_arg(format!("-I{}", pg_include))
let bindings = get_bindings(&pg_include) // Gets initial bindings that are OS-dependant
// The input header we would like to generate
// bindings for.
.header("wrapper.h")
Expand All @@ -63,6 +62,95 @@ fn main() {
println!("cargo:rustc-cfg=feature=\"{}\"", feature_version);
}

#[cfg(windows)]
fn get_bindings(pg_include: &str) -> bindgen::Builder {
// Compilation in windows requires these extra inclde paths
let pg_include_win32_msvc = format!("{}\\port\\win32_msvc", pg_include);
let pg_include_win32 = format!("{}\\port\\win32", pg_include);
// The `pg_include` path comes in the format og "includes/server", but we also need
// the parent folder, so we remove the "/server" part at the end
let pg_include_parent = pg_include[..(pg_include.len() - 7)].to_owned();

bindgen::Builder::default()
.clang_arg(format!("-I{}", pg_include_win32_msvc))
.clang_arg(format!("-I{}", pg_include_win32))
.clang_arg(format!("-I{}", pg_include))
.clang_arg(format!("-I{}", pg_include_parent))
// Whitelist all PG-related functions
.whitelist_function("pg.*")
// Whitelist used functions
.whitelist_function("longjmp")
.whitelist_function("_setjmp")
.whitelist_function("cstring_to_text")
.whitelist_function("text_to_cstring")
.whitelist_function("errmsg")
.whitelist_function("errstart")
.whitelist_function("errfinish")
.whitelist_function("pfree")
// Whitelist all PG-related types
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whitelist can limit the usefulness of pg-extend-rs; some extension modules use pg_sys to get access to additional PostgreSQL functions without needing to run bindgen themself.

Have you considered any alternatives to such whitelisting?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it can limit the usefulness, but it generates a HUGE file if you leave as default everything:

  • Linux generated file is 38k LoC
  • Windows generated file is 132k LoC

I'm not confortable with having a hard-coded list, that's why I limited only for Windows while an alternative is figured out

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we introduce a feature flag that would enable all symbols vs. limit them?

“all-symbols” seems fairly obvious. How we dice and slice the smaller sets based on which features you’re enabling is a little harder to see.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem I encountered was that "allowing all" introuced a lot of things (if I'm not wrong, it imports the full Windows API) which I started by blacklisting them, but the list is so big, that it's simpler to do a whitelisting (which works recursively) of what was used from the crate.

As I mentioned, this whitelisting based approach is only for Windows. I left the "include all" approach for the unix family

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to suggestions on how to tackle the white/black listing. One option would be to be able to pull the list of all the PG symbols and use it for the whitelisting, but I have no idea on how to do that or where to pull it from. If understand correctly how bindgen works, whitelisting that list of symbols will recursively include any non-PG symbols that the PG ones use

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if it does include the entirety of Windows API headers, what problems does that cause? Looking at bindgen's issue tracker, it seems that's a common way to use it. (e.g. rust-lang/rust-bindgen#1562 "I'm pretty sure on Firefox bindgen ends up including windows.h without issues")

You said it's slow to compile; is that the only reason why you want whitelisting? How much time does it save?

If it's merely an optimization of build speed, I don't see why this needs to be Windows-specific at all. Omitting potentially useful symbols from a module depending on platform, for only build speed reasons, seems like very surprising behavior to me, likely to cause porting issues to downstream users.

If the speed gains are worthwhile, a feature flag, as suggested by @bluejekyll, sounds like a great idea. Then Unix builds could also take advantage of it, if desired. But that would better be done as a separate PR IMO, and strip whitelisting from this PR.

Copy link
Contributor Author

@auterium auterium Dec 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comment @intgr. The reason to not include all was not due to slow compilation, but because it includes problematic code that has issues at the Rust compiler level.

error[E0588]: packed type cannot transitively contain a `[repr(align)]` type
     --> E:\rust-projects\pg-extend-rs\target\release\build\pg-extend-7be983388aa96263\out/postgres.rs:32012:1
      |
32012 | / pub struct _IMAGE_TLS_DIRECTORY64 {
32013 | |     pub StartAddressOfRawData: ULONGLONG,
32014 | |     pub EndAddressOfRawData: ULONGLONG,
32015 | |     pub AddressOfIndex: ULONGLONG,
...     |
32018 | |     pub __bindgen_anon_1: _IMAGE_TLS_DIRECTORY64__bindgen_ty_1,
32019 | | }
      | |_^

error[E0277]: arrays only have std trait implementations for lengths 0..=32
     --> E:\rust-projects\pg-extend-rs\target\release\build\pg-extend-7be983388aa96263\out/postgres.rs:51363:5
      |
51363 |     pub __bindgen_padding_0: [u8; 40usize],
      |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ the trait `std::array::LengthAtMost32` is not implemented for `[u8; 40]`
      |
      = note: required because of the requirements on the impl of `std::fmt::Debug` for `[u8; 40]`
      = note: required because of the requirements on the impl of `std::fmt::Debug` for `&[u8; 40]`
      = note: required for the cast to the object type `dyn std::fmt::Debug`

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0277`.
error: could not compile `pg-extend`.

I first tried the blacklisting approach but it was a difficult one so I moved to the whitelisting approach.

On the other hand (based on what I understood from @bluejekyll's comment, please correct me if I'm wrong), the list of whitelisted symbols could be based on feature flags, but I would still aim for the "include all" to be to whitelist all PG related things, which will recursivelly include any extra things that are required.

The fact that there's a ~130k LoC reduction was just a bonus, not the main reason for the PR

Copy link
Contributor

@intgr intgr Dec 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this issue

error[E0277]: arrays only have std trait implementations for lengths 0..=32

would be solved by updating bindgen. Our Cargo.lock file references bindgen 0.48.1 which is more than half a year old!

The other issue might be solved as well (rust-lang/rust-bindgen#1498?). Give it a shot.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I bumped bindgen to 0.51.1 and still the same errors

.whitelist_type("PG.*")
// Whitelist used types
.whitelist_type("jmp_buf")
.whitelist_type("text")
.whitelist_type("varattrib_1b")
.whitelist_type("varattrib_4b")
// Whitelist PG-related values
.whitelist_var("PG.*")
// Whitelist log-level values
.whitelist_var("DEBUG.*")
.whitelist_var("LOG.*")
.whitelist_var("INFO")
.whitelist_var("NOTICE")
.whitelist_var("WARNING")
.whitelist_var("ERROR")
.whitelist_var("FATAL")
.whitelist_var("PANIC")
// Whitelist misc values
.whitelist_var("CurrentMemoryContext")
.whitelist_var("FUNC_MAX_ARGS")
.whitelist_var("INDEX_MAX_KEYS")
.whitelist_var("NAMEDATALEN")
.whitelist_var("USE_FLOAT.*")

// FDW whitelisting
.whitelist_function("pstrdup")
.whitelist_function("lappend")
.whitelist_function("makeTargetEntry")
.whitelist_function("makeVar")
.whitelist_function("ExecStoreTuple")
.whitelist_function("heap_form_tuple")
.whitelist_function("ExecClearTuple")
.whitelist_function("slot_getallattrs")
.whitelist_function("get_rel_name")
.whitelist_function("GetForeignTable")
.whitelist_function("make_foreignscan")
.whitelist_function("extract_actual_clauses")
.whitelist_function("add_path")
.whitelist_function("create_foreignscan_path")
.whitelist_type("ImportForeignSchemaStmt")
.whitelist_type("ResultRelInfo")
.whitelist_type("EState")
.whitelist_type("ModifyTableState")
.whitelist_type("Relation")
.whitelist_type("RangeTblEntry")
.whitelist_type("Query")
.whitelist_type("ForeignScanState")
.whitelist_type("InvalidBuffer")
.whitelist_type("RelationData")
.whitelist_type("ForeignScan")
.whitelist_type("Plan")
.whitelist_type("ForeignPath")
.whitelist_type("RelOptInfo")
.whitelist_type("Form_pg_attribute")
.whitelist_var("InvalidBuffer")
}

#[cfg(unix)]
fn get_bindings(pg_include: &str) -> bindgen::Builder {
bindgen::Builder::default()
.clang_arg(format!("-I{}", pg_include))
}

fn include_dir() -> Result<String, env::VarError> {
env::var("PG_INCLUDE_PATH").or_else(|err| {
match Command::new("pg_config")
Expand Down
59 changes: 58 additions & 1 deletion pg-extend/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ pub fn register_panic_handler() {

// the panic came from a pg longjmp... so unwrap it and rethrow
unsafe {
pg_sys::siglongjmp(
pg_sys_longjmp(
pg_sys::PG_exception_stack as *mut _,
panic_context.jump_value,
);
Expand All @@ -136,6 +136,16 @@ pub fn register_panic_handler() {
}));
}

#[cfg(windows)]
unsafe fn pg_sys_longjmp(_buf: *mut pg_sys::_JBTYPE, _value: ::std::os::raw::c_int) {
pg_sys::longjmp(_buf, _value);
}

#[cfg(unix)]
unsafe fn pg_sys_longjmp(_buf: *mut pg_sys::_JBTYPE, _value: ::std::os::raw::c_int) {
pg_sys::siglongjmp(_buf, _value);
}

/// Provides a barrier between Rust and Postgres' usage of the C set/longjmp
///
/// In the case of a longjmp being caught, this will convert that to a panic. For this to work
Expand All @@ -144,6 +154,7 @@ pub fn register_panic_handler() {
/// this is already handled.
///
/// See the man pages for info on setjmp http://man7.org/linux/man-pages/man3/setjmp.3.html
#[cfg(unix)]
#[inline(never)]
pub(crate) unsafe fn guard_pg<R, F: FnOnce() -> R>(f: F) -> R {
// setup the check protection
Expand Down Expand Up @@ -179,6 +190,49 @@ pub(crate) unsafe fn guard_pg<R, F: FnOnce() -> R>(f: F) -> R {
result
}

/// Provides a barrier between Rust and Postgres' usage of the C set/longjmp
///
/// In the case of a longjmp being caught, this will convert that to a panic. For this to work
/// properly, there must be a Rust panic handler (see crate::register_panic_handler).PanicContext
/// If the `pg_exern` attribute macro is used for exposing Rust functions to Postgres, then
/// this is already handled.
///
/// See the man pages for info on setjmp http://man7.org/linux/man-pages/man3/setjmp.3.html
#[cfg(windows)]
#[inline(never)]
pub(crate) unsafe fn guard_pg<R, F: FnOnce() -> R>(f: F) -> R {
// setup the check protection
let original_exception_stack: *mut pg_sys::jmp_buf = pg_sys::PG_exception_stack;
let mut local_exception_stack: mem::MaybeUninit<pg_sys::jmp_buf> =
mem::MaybeUninit::uninit();
let jumped = pg_sys::_setjmp(
// grab a mutable reference, cast to a mutabl pointr, then case to the expected erased pointer type
local_exception_stack.as_mut_ptr() as *mut pg_sys::jmp_buf as *mut _,
);
// now that we have the local_exception_stack, we set that for any PG longjmps...

if jumped != 0 {
notice!("PG longjmped: {}", jumped);
pg_sys::PG_exception_stack = original_exception_stack;

// The C Panicked!, handling control to Rust Panic handler
compiler_fence(Ordering::SeqCst);
panic!(JumpContext { jump_value: jumped });
}

// replace the exception stack with ours to jump to the above point
pg_sys::PG_exception_stack = local_exception_stack.as_mut_ptr() as *mut _;

// enforce that the setjmp is not reordered, though that's probably unlikely...
compiler_fence(Ordering::SeqCst);
let result = f();

compiler_fence(Ordering::SeqCst);
pg_sys::PG_exception_stack = original_exception_stack;

result
}

/// auto generate function to output a SQL create statement for the function
///
/// Until concat_ident! stabilizes, this requires the name to passed with the appended sctring
Expand Down Expand Up @@ -214,6 +268,9 @@ macro_rules! pg_create_stmt_bin {
#[cfg(target_os = "macos")]
const DYLIB_EXT: &str = "dylib";

#[cfg(target_os = "windows")]
const DYLIB_EXT: &str = "dll";

fn main() {
const LIB_NAME: &str = env!("CARGO_PKG_NAME");

Expand Down