diff --git a/README.md b/README.md index b1b98d41e..af3b33c33 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ ![Matrix-icon](assets/img/readme/header.png) [![license](http://img.shields.io/badge/license-BSD3-brightgreen.svg?style=flat)](https://github.com/Tencent/matrix/blob/master/LICENSE) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/Tencent/matrix/pulls) -[![WeChat Approved](https://img.shields.io/badge/Wechat%20Approved-2.0.1-red.svg)](https://github.com/Tencent/matrix/wiki) +[![WeChat Approved](https://img.shields.io/badge/Wechat%20Approved-2.0.2-red.svg)](https://github.com/Tencent/matrix/wiki) [![CircleCI](https://circleci.com/gh/Tencent/matrix.svg?style=shield)](https://app.circleci.com/pipelines/github/Tencent/matrix) (中文版本请参看[这里](#matrix_cn)) @@ -140,6 +140,10 @@ At this point, Matrix has been integrated into the app and is beginning to colle - **Battery Canary:** App thread activities monitor (Background watch & foreground loop watch), Sonsor usage monitor (WakeLock/Alarm/Gps/Wifi/Bluetooth), Background network activities (Wifi/Mobile) monitor. + +- **MemGuard** + + Detect heap memory overlap, use-after-free and double free issues. ## Features @@ -197,6 +201,14 @@ At this point, Matrix has been integrated into the app and is beginning to colle + **Non-invasive.** It is based on PLT-hook([iqiyi/xHook](https://github.com/iqiyi/xHook)), so we do NOT need to recompile the native libraries. + WebView still works after using this tool. +#### MemGuard + ++ A tool base on GWP-Asan to detect heap memory issues. ++ **Non-invasive.** It is based on PLT-hook([iqiyi/xHook](https://github.com/iqiyi/xHook)), so we do NOT need to recompile the native libraries. ++ It's able to apply on specific libraries that needs to be detected by RegEx. + ++ It detects heap memory accessing overlap, use-after-free and double free issues. + #### Backtrace Component @@ -208,7 +220,7 @@ At this point, Matrix has been integrated into the app and is beginning to colle 1. Configure `MATRIX_VERSION` in gradle.properties. ``` gradle - MATRIX_VERSION=2.0.1 + MATRIX_VERSION=2.0.2 ``` 2. Add `matrix-gradle-plugin` in your build.gradle: @@ -357,11 +369,11 @@ Then other components in Matrix could use Quikcen Backtrace to unwind stacktrace #### APK Checker Usage -APK Checker can run independently in Jar ([matrix-apk-canary-2.0.1.jar](https://repo.maven.apache.org/maven2/com/tencent/matrix/matrix-apk-canary/2.0.1/matrix-apk-canary-2.0.1.jar)) mode, usage: +APK Checker can run independently in Jar ([matrix-apk-canary-2.0.2.jar](https://repo.maven.apache.org/maven2/com/tencent/matrix/matrix-apk-canary/2.0.2/matrix-apk-canary-2.0.2.jar)) mode, usage: ```shell -java -jar matrix-apk-canary-2.0.1.jar +java -jar matrix-apk-canary-2.0.2.jar Usages: --config CONFIG-FILE-PATH or @@ -420,7 +432,7 @@ Matrix is under the BSD license. See the [LICENSE](https://github.com/Tencent/Ma # Matrix ![Matrix-icon](assets/img/readme/header.png) -[![license](http://img.shields.io/badge/license-BSD3-brightgreen.svg?style=flat)](https://github.com/Tencent/matrix/blob/master/LICENSE)[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/Tencent/matrix/pulls) [![WeChat Approved](https://img.shields.io/badge/Wechat%20Approved-2.0.1-red.svg)](https://github.com/Tencent/matrix/wiki) +[![license](http://img.shields.io/badge/license-BSD3-brightgreen.svg?style=flat)](https://github.com/Tencent/matrix/blob/master/LICENSE)[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/Tencent/matrix/pulls) [![WeChat Approved](https://img.shields.io/badge/Wechat%20Approved-2.0.2-red.svg)](https://github.com/Tencent/matrix/wiki) **Matrix** 是一款微信研发并日常使用的应用性能接入框架,支持iOS, macOS和Android。 Matrix 通过接入各种性能监控方案,对性能监控项的异常数据进行采集和分析,输出相应的问题分析、定位与优化建议,从而帮助开发者开发出更高质量的应用。 @@ -529,16 +541,25 @@ curBuilder.pluginListener = <一个遵循 MatrixPluginListenerDelegate 的对象 Matrix-android 当前监控范围包括:应用安装包大小,帧率变化,启动耗时,卡顿,慢方法,SQLite 操作优化,文件读写,内存泄漏等等。 - APK Checker: 针对 APK 安装包的分析检测工具,根据一系列设定好的规则,检测 APK 是否存在特定的问题,并输出较为详细的检测结果报告,用于分析排查问题以及版本追踪 + - Resource Canary: 基于 WeakReference 的特性和 [Square Haha](https://github.com/square/haha) 库开发的 Activity 泄漏和 Bitmap 重复创建检测工具 + - Trace Canary: 监控ANR、界面流畅性、启动耗时、页面切换耗时、慢函数及卡顿等问题 + - SQLite Lint: 按官方最佳实践自动化检测 SQLite 语句的使用质量 + - IO Canary: 检测文件 IO 问题,包括:文件 IO 监控和 Closeable Leak 监控 + - Battery Canary: 监控 App 活跃线程(待机状态 & 前台 Loop 监控)、ASM 调用 (WakeLock/Alarm/Gps/Wifi/Bluetooth 等传感器)、 后台流量 (Wifi/移动网络)等 Battery Historian 统计 App 耗电的数据 + +- MemGuard + + 检测堆内存访问越界、使用释放后的内存、重复释放等问题 ## 特性 @@ -599,6 +620,13 @@ Matrix-android 当前监控范围包括:应用安装包大小,帧率变化 + 无侵入,基于 PLT-hook([iqiyi/xHook](https://github.com/iqiyi/xHook)),无需重编 native 库 + 使用该工具后 WebView 仍可正常工作 +#### MemGuard + ++ 一个基于 GWP-Asan 修改的堆内存问题检测工具 ++ 无侵入,基于 PLT-hook([iqiyi/xHook](https://github.com/iqiyi/xHook)),无需重编 native 库 ++ 可根据正则表达式指定被检测的目标库 ++ 可检测堆内存访问越界、使用释放后的内存和双重释放等问题 + #### Backtrace Component - 基于 DWARF 以及 ARM 异常处理数据进行简化并生成全新的 quicken unwind tables 数据,用于实现可快速回溯 native 调用栈的 backtrace 组件。回溯速度约是 libunwindstack 的 15x ~ 30x 左右。 @@ -608,7 +636,7 @@ Matrix-android 当前监控范围包括:应用安装包大小,帧率变化 1. 在你项目根目录下的 gradle.properties 中配置要依赖的 Matrix 版本号,如: ``` gradle - MATRIX_VERSION=2.0.1 + MATRIX_VERSION=2.0.2 ``` 2. 在你项目根目录下的 build.gradle 文件添加 Matrix 依赖,如: @@ -754,10 +782,10 @@ WeChatBacktrace.instance().configure(getApplicationContext()).commit(); #### APK Checker -APK Check 以独立的 jar 包提供 ([matrix-apk-canary-2.0.1.jar](https://repo.maven.apache.org/maven2/com/tencent/matrix/matrix-apk-canary/2.0.1/matrix-apk-canary-2.0.1.jar)),你可以运行: +APK Check 以独立的 jar 包提供 ([matrix-apk-canary-2.0.2.jar](https://repo.maven.apache.org/maven2/com/tencent/matrix/matrix-apk-canary/2.0.2/matrix-apk-canary-2.0.2.jar)),你可以运行: ```cmd -java -jar matrix-apk-canary-2.0.1.jar +java -jar matrix-apk-canary-2.0.2.jar ``` 查看 Usages 来使用它。 diff --git a/matrix/matrix-android/gradle.properties b/matrix/matrix-android/gradle.properties index 55da6a647..97a045182 100644 --- a/matrix/matrix-android/gradle.properties +++ b/matrix/matrix-android/gradle.properties @@ -14,7 +14,7 @@ org.gradle.jvmargs=-Xmx2048m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryErro # org.gradle.parallel=true #Tue Jun 20 10:24:33 CST 2017 -VERSION_NAME_PREFIX=2.0.1 +VERSION_NAME_PREFIX=2.0.2 VERSION_NAME_SUFFIX= ## two options: Internal (for wechat), External (for public repo) PUBLISH_CHANNEL=Internal diff --git a/matrix/matrix-android/matrix-apk-canary/src/main/java/com/tencent/matrix/apk/model/task/UnzipTask.java b/matrix/matrix-android/matrix-apk-canary/src/main/java/com/tencent/matrix/apk/model/task/UnzipTask.java index 55931057f..409f36dcd 100644 --- a/matrix/matrix-android/matrix-apk-canary/src/main/java/com/tencent/matrix/apk/model/task/UnzipTask.java +++ b/matrix/matrix-android/matrix-apk-canary/src/main/java/com/tencent/matrix/apk/model/task/UnzipTask.java @@ -227,11 +227,16 @@ private String reverseResguard(String dirName, String filename) { private String writeEntry(ZipFile zipFile, ZipEntry entry) throws IOException { + String entryName = entry.getName(); + if (Util.preventZipSlip(outputFile, entryName)) { + Log.e(TAG, "writeEntry entry %s failed!", entryName); + return null; + } + int readSize; byte[] readBuffer = new byte[4096]; BufferedOutputStream bufferedOutput = null; InputStream zipInputStream = null; - String entryName = entry.getName(); String outEntryName = null; String filename; File dir; diff --git a/matrix/matrix-android/matrix-backtrace/build.gradle b/matrix/matrix-android/matrix-backtrace/build.gradle index acd038f33..6d52c9cd7 100644 --- a/matrix/matrix-android/matrix-backtrace/build.gradle +++ b/matrix/matrix-android/matrix-backtrace/build.gradle @@ -19,6 +19,7 @@ android { externalNativeBuild { cmake { + targets = ['wechatbacktrace', 'unwindstack'] arguments = ['-DANDROID_STL=c++_shared', "-DEnableLOG=${gradle.enableLog() ? "ON" : "OFF"}" as String, "-DQUT_STATISTIC_ENABLE=${gradle.enableLog() ? "ON" : "OFF"}" as String] diff --git a/matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/CMakeLists.txt b/matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/CMakeLists.txt index 22aaa2ec0..c0e651101 100755 --- a/matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/CMakeLists.txt +++ b/matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/CMakeLists.txt @@ -17,6 +17,7 @@ SET( ${CMAKE_CURRENT_SOURCE_DIR}/ElfInterfaceArm.cpp ${CMAKE_CURRENT_SOURCE_DIR}/JitDebug.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Log.cpp + ${CMAKE_CURRENT_SOURCE_DIR}/LocalUnwinder.cpp ${CMAKE_CURRENT_SOURCE_DIR}/MapInfo.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Maps.cpp ${CMAKE_CURRENT_SOURCE_DIR}/Memory.cpp diff --git a/matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/LocalUnwinder.cpp b/matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/LocalUnwinder.cpp index 05650fb2e..e48aaac91 100644 --- a/matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/LocalUnwinder.cpp +++ b/matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/LocalUnwinder.cpp @@ -31,7 +31,9 @@ #include #include +#include #include +#include #include #include @@ -142,4 +144,105 @@ bool LocalUnwinder::Unwind(std::vector* frame_info, size_t max_f return num_frames != 0; } +size_t LocalUnwinder::UnwindImpl(std::unique_ptr& regs, void** pcs, size_t max_frames) { + ArchEnum arch = regs->Arch(); + + size_t num_frames = 0; + bool adjust_pc = false; + while (true) { + uint64_t cur_pc = regs->pc(); + uint64_t cur_sp = regs->sp(); + + MapInfo* map_info = GetMapInfo(cur_pc); + if (map_info == nullptr) { + break; + } + + Elf* elf = map_info->GetElf(process_memory_, arch); + uint64_t rel_pc = elf->GetRelPc(cur_pc, map_info); + uint64_t step_pc = rel_pc; + uint64_t pc_adjustment; + if (adjust_pc) { + pc_adjustment = GetPcAdjustment(rel_pc, elf, arch); + } else { + pc_adjustment = 0; + } + step_pc -= pc_adjustment; + + bool finished = false; + if (elf->StepIfSignalHandler(rel_pc, regs.get(), process_memory_.get())) { + step_pc = rel_pc; + } else if (!elf->Step(step_pc, regs.get(), process_memory_.get(), &finished)) { + finished = true; + } + + // Skip any locations that are within this library. + if (num_frames != 0 || !ShouldSkipLibrary(map_info->name)) { + pcs[num_frames++] = (void*) cur_pc; + if (num_frames >= max_frames) { + break; + } + } + + if (finished || num_frames == max_frames || + (cur_pc == regs->pc() && cur_sp == regs->sp())) { + break; + } + adjust_pc = true; + } + return num_frames; +} + +size_t LocalUnwinder::Unwind(void** pcs, size_t max_frames) { + std::unique_ptr regs(unwindstack::Regs::CreateFromLocal()); + unwindstack::RegsGetLocal(regs.get()); + return UnwindImpl(regs, pcs, max_frames); +} + +size_t LocalUnwinder::Unwind(void* ucontext, void** pcs, size_t max_frames) { + std::unique_ptr regs(unwindstack::Regs::CreateFromUcontext(Regs::CurrentArch(), ucontext)); + return UnwindImpl(regs, pcs, max_frames); +} + +extern "C" char* __cxa_demangle(const char*, char*, size_t*, int* status); + +bool LocalUnwinder::GetStackElementString(uint64_t pc, std::string* string_out) { + MapInfo* map_info = GetMapInfo(pc); + if (map_info == nullptr) { + return false; + } + auto elf = map_info->GetElf(process_memory_, Regs::CurrentArch()); + uint64_t rel_pc = elf->GetRelPc(pc, map_info); + + std::stringstream ss; + std::ios fmt(nullptr); + fmt.copyfmt(ss); + ss << "pc " << std::hex << std::setw(sizeof(void*) * 2) << std::setfill('0') << rel_pc << std::dec << ' '; + ss.copyfmt(fmt); + + std::string funcName; + uint64_t funcOffset = 0; + bool isFuncNameGot = map_info->GetFunctionName(rel_pc, &funcName, &funcOffset); + if (isFuncNameGot) { + char* demangledFuncName = nullptr; + if (!funcName.empty()) { + demangledFuncName = __cxa_demangle(funcName.c_str(), nullptr, nullptr, nullptr); + } + if (demangledFuncName != nullptr) { + ss << map_info->name << " (" << demangledFuncName << ")"; + free(demangledFuncName); + } else { + ss << map_info->name << " (" << funcName << ")"; + } + } else { + ss << map_info->name << " (?\?\?)"; + } + auto buildID = map_info->GetPrintableBuildID(); + if (!buildID.empty()) { + ss << " (BuildID: " << map_info->GetPrintableBuildID() << ")"; + } + *string_out = ss.str(); + return true; +} + } // namespace unwindstack diff --git a/matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/include/unwindstack/LocalUnwinder.h b/matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/include/unwindstack/LocalUnwinder.h index 80bb53ec1..f69f953bd 100644 --- a/matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/include/unwindstack/LocalUnwinder.h +++ b/matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/include/unwindstack/LocalUnwinder.h @@ -66,14 +66,22 @@ class LocalUnwinder { bool Unwind(std::vector* frame_info, size_t max_frames); + size_t Unwind(void** pcs, size_t max_frames); + + size_t Unwind(void* ucontext, void** pcs, size_t max_frames); + bool ShouldSkipLibrary(const std::string& map_name); MapInfo* GetMapInfo(uint64_t pc); + bool GetStackElementString(uint64_t pc, std::string* string_out); + ErrorCode LastErrorCode() { return last_error_.code; } uint64_t LastErrorAddress() { return last_error_.address; } private: + size_t UnwindImpl(std::unique_ptr& regs, void** pcs, size_t max_frames); + pthread_rwlock_t maps_rwlock_; std::unique_ptr maps_ = nullptr; std::shared_ptr process_memory_; diff --git a/matrix/matrix-android/matrix-backtrace/src/main/cpp/libwechatbacktrace/Dummy.cpp b/matrix/matrix-android/matrix-backtrace/src/main/cpp/libwechatbacktrace/Dummy.cpp new file mode 100644 index 000000000..c24f77fc2 --- /dev/null +++ b/matrix/matrix-android/matrix-backtrace/src/main/cpp/libwechatbacktrace/Dummy.cpp @@ -0,0 +1,4 @@ +// +// Created by YinSheng Tang on 2021/9/23. +// + diff --git a/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpDelegate.java b/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpDelegate.java index 4836a153a..6b5196f31 100644 --- a/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpDelegate.java +++ b/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpDelegate.java @@ -27,6 +27,7 @@ import android.os.Message; import android.os.OperationCanceledException; import android.os.Process; +import android.system.ErrnoException; import android.system.Os; import android.system.StructStat; import android.util.Pair; @@ -518,8 +519,13 @@ public void run() { iterateTargetDirectory(file, cs, new FileFilter() { @Override public boolean accept(File pathname) { - count[0] += 1; - count[1] += pathname.isFile() ? pathname.length() : 0; + try { + StructStat stat = Os.lstat(pathname.getAbsolutePath()); + count[0] += 1; + count[1] += stat.st_blocks * stat.st_blksize; + } catch (ErrnoException e) { + MatrixLog.printErrStackTrace(TAG, e, ""); + } return false; } }); diff --git a/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpScheduler.java b/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpScheduler.java index 6ae88e902..34ce4144e 100644 --- a/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpScheduler.java +++ b/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpScheduler.java @@ -197,10 +197,10 @@ private final static class IdleReceiver extends BroadcastReceiver { Handler mIdleHandler; Context mContext; - private WeChatBacktrace.WarmUpTiming mTiming; - private long mWarmUpDelay; + private final WeChatBacktrace.WarmUpTiming mTiming; + private final long mWarmUpDelay; - private Set mTasks = new HashSet<>(); + private final Set mTasks = new HashSet<>(); IdleReceiver(Context context, Handler idleHandler, WeChatBacktrace.WarmUpTiming timing, long delay) { diff --git a/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpUtility.java b/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpUtility.java index 252a32b1b..721a46e17 100644 --- a/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpUtility.java +++ b/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpUtility.java @@ -41,10 +41,10 @@ class WarmUpUtility { private final static String FILE_BLOCKED_LIST = "blocked-list"; private final static String FILE_UNFINISHED = "unfinished"; - final static long DURATION_LAST_ACCESS_FAR_FUTURE = 30L * 24 * 3600 * 1000; // milliseconds - final static long DURATION_LAST_ACCESS_EXPIRED = 60L * 24 * 3600 * 1000; // milliseconds - final static long DURATION_CLEAN_UP_EXPIRED = 3L * 24 * 3600 * 1000; // milliseconds - final static long DURATION_CLEAN_UP = 3L * 24 * 3600 * 1000; // milliseconds + final static long DURATION_LAST_ACCESS_FAR_FUTURE = 7L * 24 * 3600 * 1000; // milliseconds + final static long DURATION_LAST_ACCESS_EXPIRED = 3L * 24 * 3600 * 1000; // milliseconds + final static long DURATION_CLEAN_UP_EXPIRED = 2L * 24 * 3600 * 1000; // milliseconds + final static long DURATION_CLEAN_UP = 2L * 24 * 3600 * 1000; // milliseconds final static long DURATION_DISK_USAGE_COMPUTATION = 3L * 24 * 3600 * 1000; // milliseconds final static int WARM_UP_FILE_MAX_RETRY = 3; @@ -61,8 +61,7 @@ private static int retryCount(Context context, String key) { mUnfinishedWarmUp = WarmUpUtility.readUnfinishedMaps(context); } Integer value = mUnfinishedWarmUp.get(key); - int retryCount = value != null ? value : 0; - return retryCount; + return value != null ? value : 0; } public static boolean check(Context context, String pathOfElf, int offset) { diff --git a/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WeChatBacktrace.java b/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WeChatBacktrace.java index fa68ae5e7..7ff3801d7 100644 --- a/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WeChatBacktrace.java +++ b/matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WeChatBacktrace.java @@ -75,8 +75,8 @@ public static String getBaseODEXPath(Context context) { private volatile boolean mInitialized; private volatile boolean mConfigured; private volatile Configuration mConfiguration; - private WarmUpDelegate mWarmUpDelegate = new WarmUpDelegate(); - private Handler mHandler = new Handler(Looper.getMainLooper()); + private final WarmUpDelegate mWarmUpDelegate = new WarmUpDelegate(); + private final Handler mHandler = new Handler(Looper.getMainLooper()); private static boolean sLibraryLoaded = false; @@ -345,7 +345,7 @@ public final static class Configuration { String mPathOfXLogSo = null; private boolean mCommitted = false; - private WeChatBacktrace mWeChatBacktrace; + private final WeChatBacktrace mWeChatBacktrace; Configuration(Context context, WeChatBacktrace backtrace) { mContext = context; diff --git a/matrix/matrix-android/matrix-battery-canary/build.gradle b/matrix/matrix-android/matrix-battery-canary/build.gradle index f67f86f21..4b5f2a0ec 100644 --- a/matrix/matrix-android/matrix-battery-canary/build.gradle +++ b/matrix/matrix-android/matrix-battery-canary/build.gradle @@ -36,6 +36,7 @@ dependencies { testImplementation "org.mockito:mockito-core:2.8.9" testImplementation 'org.jmockit:jmockit:1.28' testImplementation 'com.google.code.gson:gson:2.8.6' + androidTestImplementation 'commons-io:commons-io:2.6' androidTestImplementation 'androidx.core:core:1.3.2' androidTestImplementation 'androidx.annotation:annotation:1.0.0' androidTestImplementation 'androidx.test.ext:junit:1.1.1' diff --git a/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/monitor/feature/MonitorFeatureOverAllTest.java b/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/monitor/feature/MonitorFeatureOverAllTest.java index d65d18eb4..936a9a91e 100644 --- a/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/monitor/feature/MonitorFeatureOverAllTest.java +++ b/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/monitor/feature/MonitorFeatureOverAllTest.java @@ -65,6 +65,7 @@ private BatteryMonitorCore mockMonitor() { .enable(AlarmMonitorFeature.class) .enable(AppStatMonitorFeature.class) .enable(BlueToothMonitorFeature.class) + .enable(CpuStatFeature.class) .enableBuiltinForegroundNotify(false) .enableForegroundMode(true) .wakelockTimeout(1000) diff --git a/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/monitor/feature/StatFeatureCpuTest.java b/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/monitor/feature/StatFeatureCpuTest.java new file mode 100644 index 000000000..e15ded4b0 --- /dev/null +++ b/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/monitor/feature/StatFeatureCpuTest.java @@ -0,0 +1,193 @@ +/* + * Tencent is pleased to support the open source community by making wechat-matrix available. + * Copyright (C) 2018 THL A29 Limited, a Tencent company. All rights reserved. + * Licensed under the BSD 3-Clause License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.tencent.matrix.batterycanary.monitor.feature; + +import android.app.Application; +import android.content.Context; + +import com.tencent.matrix.Matrix; +import com.tencent.matrix.batterycanary.monitor.BatteryMonitorConfig; +import com.tencent.matrix.batterycanary.monitor.BatteryMonitorCore; +import com.tencent.matrix.batterycanary.monitor.feature.CpuStatFeature.CpuStateSnapshot; +import com.tencent.matrix.batterycanary.monitor.feature.MonitorFeature.Snapshot.Delta; +import com.tencent.matrix.batterycanary.monitor.feature.MonitorFeature.Snapshot.Entry.DigitEntry; +import com.tencent.matrix.batterycanary.utils.BatteryMetricsTest; +import com.tencent.matrix.batterycanary.utils.PowerProfile; +import com.tencent.matrix.batterycanary.utils.ProcStatUtil; + +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +import androidx.test.ext.junit.runners.AndroidJUnit4; +import androidx.test.platform.app.InstrumentationRegistry; + + +@RunWith(AndroidJUnit4.class) +public class StatFeatureCpuTest { + static final String TAG = "Matrix.test.StatFeatureCpuTest"; + + Context mContext; + + @Before + public void setUp() { + mContext = InstrumentationRegistry.getInstrumentation().getTargetContext(); + if (!Matrix.isInstalled()) { + Matrix.init(new Matrix.Builder(((Application) mContext.getApplicationContext())).build()); + } + } + + @After + public void shutDown() { + } + + private BatteryMonitorCore mockMonitor() { + BatteryMonitorConfig config = new BatteryMonitorConfig.Builder() + .enable(CpuStatFeature.class) + .enableBuiltinForegroundNotify(false) + .enableForegroundMode(false) + .wakelockTimeout(1000) + .greyJiffiesTime(100) + .foregroundLoopCheckTime(1000) + .build(); + return new BatteryMonitorCore(config); + } + + @Test + public void testGetCpuStatesSnapshot() throws InterruptedException { + CpuStatFeature feature = new CpuStatFeature(); + feature.configure(mockMonitor()); + feature.onTurnOn(); + + Assert.assertTrue(feature.isSupported()); + CpuStateSnapshot cpuStateSnapshot = feature.currentCpuStateSnapshot(); + Assert.assertFalse(cpuStateSnapshot.isDelta); + Assert.assertTrue(cpuStateSnapshot.isValid()); + Assert.assertTrue(cpuStateSnapshot.totalCpuJiffies() > 0); + Assert.assertTrue(cpuStateSnapshot.cpuCoreStates.size() > 0); + Assert.assertTrue(cpuStateSnapshot.procCpuCoreStates.size() > 0); + } + + @Test + public void testGetCpuStatesSnapshotDelta() throws InterruptedException { + CpuStatFeature feature = new CpuStatFeature(); + feature.configure(mockMonitor()); + feature.onTurnOn(); + + Assert.assertTrue(feature.isSupported()); + + CpuStateSnapshot bgn = feature.currentCpuStateSnapshot(); + Thread.sleep(1000L); + CpuStateSnapshot end = feature.currentCpuStateSnapshot(); + Delta delta = end.diff(bgn); + + Assert.assertTrue(delta.bgn.isValid()); + Assert.assertTrue(delta.end.isValid()); + Assert.assertTrue(delta.dlt.isValid()); + Assert.assertTrue(delta.dlt.isDelta); + Assert.assertTrue(delta.end.totalCpuJiffies() >= delta.bgn.totalCpuJiffies()); + Assert.assertEquals(delta.dlt.totalCpuJiffies(), end.totalCpuJiffies() - bgn.totalCpuJiffies()); + Assert.assertTrue(delta.dlt.cpuCoreStates.size() > 0); + Assert.assertTrue(delta.dlt.procCpuCoreStates.size() > 0); + } + + @Test + public void testConfigureSipping() throws InterruptedException, IOException { + CpuStatFeature feature = new CpuStatFeature(); + feature.configure(mockMonitor()); + feature.onTurnOn(); + + Assert.assertTrue(feature.isSupported()); + CpuStateSnapshot cpuStateSnapshot = feature.currentCpuStateSnapshot(); + Assert.assertFalse(cpuStateSnapshot.isDelta); + Assert.assertTrue(cpuStateSnapshot.isValid()); + Assert.assertTrue(cpuStateSnapshot.totalCpuJiffies() > 0); + + PowerProfile powerProfile = PowerProfile.init(mContext); + Assert.assertNotNull(powerProfile); + Assert.assertTrue(powerProfile.isSupported()); + + double cpuSip = cpuStateSnapshot.configureCpuSip(powerProfile); + Assert.assertTrue(cpuSip > 0); + + double procSip = cpuStateSnapshot.configureProcSip(powerProfile, ProcStatUtil.currentPid().getJiffies()); + Assert.assertTrue(procSip > 0); + } + + @Test + public void testConfigureSippingDelta() throws InterruptedException, IOException { + CpuStatFeature feature = new CpuStatFeature(); + feature.configure(mockMonitor()); + feature.onTurnOn(); + + Assert.assertTrue(feature.isSupported()); + ProcStatUtil.ProcStat procStatBgn = ProcStatUtil.currentPid(); + CpuStateSnapshot bgn = feature.currentCpuStateSnapshot(); + BatteryMetricsTest.CpuConsumption.hanoi(20); + CpuStateSnapshot end = feature.currentCpuStateSnapshot(); + + for (int i = 0; i < end.procCpuCoreStates.size(); i++) { + DigitEntry entry = end.procCpuCoreStates.get(i).getList().get(0); + end.procCpuCoreStates.get(i).getList().set(0, DigitEntry.of(entry.get() + 100L * (i + 1))); + } + Delta delta = end.diff(bgn); + + Assert.assertTrue(delta.bgn.isValid()); + Assert.assertTrue(delta.end.isValid()); + Assert.assertTrue(delta.dlt.isValid()); + Assert.assertTrue(delta.dlt.isDelta); + + PowerProfile powerProfile = PowerProfile.init(mContext); + Assert.assertNotNull(powerProfile); + Assert.assertTrue(powerProfile.isSupported()); + + double cpuSip = delta.dlt.configureCpuSip(powerProfile); + Assert.assertTrue(cpuSip > 0); + + double procSip = delta.dlt.configureProcSip(powerProfile, ProcStatUtil.currentPid().getJiffies() - procStatBgn.getJiffies()); + Assert.assertTrue(procSip > 0); + } + + @Test + public void testConcurrent() throws InterruptedException { + final CpuStatFeature feature = new CpuStatFeature(); + feature.configure(mockMonitor()); + feature.onTurnOn(); + + List threadList = new ArrayList<>(); + for (int i = 0; i < 100; i++) { + final int finalI = i; + Thread thread = new Thread(new Runnable() { + @Override + public void run() { + feature.currentCpuStateSnapshot(); + } + }); + thread.start(); + threadList.add(thread); + } + for (Thread item : threadList) { + item.join(); + } + } +} diff --git a/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/BatteryMetricsTest.java b/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/BatteryMetricsTest.java new file mode 100644 index 000000000..7c9fa89cb --- /dev/null +++ b/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/BatteryMetricsTest.java @@ -0,0 +1,838 @@ +/* + * Tencent is pleased to support the open source community by making wechat-matrix available. + * Copyright (C) 2018 THL A29 Limited, a Tencent company. All rights reserved. + * Licensed under the BSD 3-Clause License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.tencent.matrix.batterycanary.utils; + +import android.content.Context; +import android.content.res.XmlResourceParser; +import android.os.FileObserver; +import android.os.Handler; +import android.os.Looper; +import android.os.Process; +import android.os.SystemClock; +import android.text.TextUtils; +import android.util.Log; + +import com.tencent.matrix.batterycanary.TestUtils; + +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.xmlpull.v1.XmlPullParser; + +import java.io.BufferedReader; +import java.io.File; +import java.io.FileReader; +import java.io.IOException; +import java.lang.reflect.Constructor; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Stack; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; + +import androidx.annotation.Nullable; +import androidx.test.ext.junit.runners.AndroidJUnit4; +import androidx.test.platform.app.InstrumentationRegistry; + +import static com.tencent.matrix.batterycanary.utils.BatteryCanaryUtil.JIFFY_MILLIS; +import static com.tencent.matrix.batterycanary.utils.BatteryCanaryUtil.ONE_HOR; +import static com.tencent.matrix.batterycanary.utils.BatteryCanaryUtil.ONE_MIN; + + +@RunWith(AndroidJUnit4.class) +public class BatteryMetricsTest { + static final String TAG = "Matrix.test.metrics"; + + Context mContext; + + @Before + public void setUp() { + mContext = InstrumentationRegistry.getInstrumentation().getTargetContext(); + } + + @After + public void shutDown() { + } + + @Test + public void testReadPowerProfile() throws Exception { + Class clazz = Class.forName("com.android.internal.os.PowerProfile"); + Constructor constructor = clazz.getConstructor(Context.class); + Object obj = constructor.newInstance(mContext); + Assert.assertNotNull(obj); + + int id = mContext.getResources().getIdentifier("power_profile", "xml", "android"); + Assert.assertTrue(id > 0); + + try (XmlResourceParser parser = mContext.getResources().getXml(id)) { + while (parser.getEventType() != XmlPullParser.END_DOCUMENT) { + if (parser.getEventType() == XmlPullParser.START_TAG) { + String tagName = parser.getName(); + String tagText = parser.getText(); + for (int i = 0; i < parser.getAttributeCount(); i++) { + String attrName = parser.getAttributeName(i); + String attrValue = parser.getAttributeValue(i); + if (attrValue.startsWith("cpu.core_speeds.cluster")) { + } + } + } + parser.nextToken(); + } + } + + PowerProfile powerProfile = PowerProfile.init(mContext); + Assert.assertNotNull(powerProfile); + Assert.assertNotNull(PowerProfile.getInstance()); + Assert.assertTrue(PowerProfile.getInstance().isSupported()); + } + + @Test + public void testGetAppCpuCoreNumByTid() { + StringBuilder tips = new StringBuilder("\n"); + int num = configureCpuCoreNum(Process.myTid()); + tips.append(Thread.currentThread().getName()).append("(").append(Process.myTid()).append(") is running on cpu" + num).append("\n"); + + String dirPath = "/proc/" + Process.myPid() + "/task"; + for (File item : new File(dirPath).listFiles()) { + if (item.isDirectory()) { + int tid = Integer.parseInt(item.getName()); + ProcStatUtil.ProcStat stat = ProcStatUtil.of(Process.myPid(), tid); + num = configureCpuCoreNum(tid); + tips.append(stat.comm).append("(").append(tid).append(") is running on cpu" + num).append("\n"); + } + } + + if (!TestUtils.isAssembleTest()) { + Assert.fail(tips.toString()); + } + } + + @Test + public void testCpuCoreChangingWithRunningThread() throws InterruptedException { + StringBuilder tips = new StringBuilder("\n"); + final AtomicInteger tid = new AtomicInteger(-1); + new Thread(new Runnable() { + @Override + public void run() { + tid.set(Process.myTid()); + while (true) {} + } + }, "TEST_THREAD").start(); + + while (tid.get() < 0) {} + for (int i = 0; i < 100; i++) { + int num = configureCpuCoreNum(tid.get()); + tips.append(num).append("\n"); + Thread.sleep(100L); + } + + if (!TestUtils.isAssembleTest()) { + Assert.fail(tips.toString()); + } + } + + @Test + public void testCpuCoreChangingWithBlockThread() throws InterruptedException { + StringBuilder tips = new StringBuilder("\n"); + final AtomicInteger tid = new AtomicInteger(-1); + new Thread(new Runnable() { + @Override + public void run() { + tid.set(Process.myTid()); + synchronized (tid) { + try { + tid.wait(); + } catch (InterruptedException e) { + throw new RuntimeException(e); + } + } + } + }, "TEST_THREAD").start(); + + while (tid.get() < 0) {} + for (int i = 0; i < 100; i++) { + int num = configureCpuCoreNum(tid.get()); + tips.append(num).append("\n"); + Thread.sleep(100L); + } + + if (!TestUtils.isAssembleTest()) { + Assert.fail(tips.toString()); + } + } + + @Test + public void testCpuCoreChangingWithUIThread() throws InterruptedException { + StringBuilder tips = new StringBuilder("\n"); + final AtomicInteger tid = new AtomicInteger(-1); + final Handler handler = new Handler(Looper.getMainLooper()); + handler.post(new Runnable() { + @Override + public void run() { + tid.set(Process.myTid()); + } + }); + + handler.post(new Runnable() { + @Override + public void run() { + try { + Thread.sleep(50L); + } catch (InterruptedException ignored) { + } + handler.post(this); + } + }); + + while (tid.get() < 0) {} + for (int i = 0; i < 100; i++) { + int num = configureCpuCoreNum(tid.get()); + tips.append(num).append("\n"); + Thread.sleep(100L); + } + + if (!TestUtils.isAssembleTest()) { + Assert.fail(tips.toString()); + } + } + + private static int configureCpuCoreNum(int tid) { + String cat = BatteryCanaryUtil.cat("proc/" + Process.myPid() + "/task/" + tid + "/stat"); + Assert.assertFalse(TextUtils.isEmpty(cat)); + int index = cat.indexOf(")"); + if (index <= 0) throw new IllegalStateException(cat + " has not ')'"); + String suffix = cat.substring(index + ")".length()); + String[] splits = suffix.split(" "); + int columnIndex = 37; + Assert.assertTrue(splits.length >= columnIndex + 1); + Assert.assertTrue(ProcStatUtil.isNumeric(splits[columnIndex])); + Assert.assertTrue(splits[columnIndex] + ": tid = " + tid, TextUtils.isDigitsOnly(splits[columnIndex])); + return Integer.parseInt(splits[columnIndex]); + } + + @Test + public void testReadKernelCpuSpeedState() throws IOException { + for (int i = 0; i < BatteryCanaryUtil.getCpuCoreNum(); i++) { + String path = "/sys/devices/system/cpu/cpu" + i + "/cpufreq/stats/time_in_state"; + try (BufferedReader reader = new BufferedReader(new FileReader(new File(path)))) { + TextUtils.SimpleStringSplitter splitter = new TextUtils.SimpleStringSplitter(' '); + String line; + while ((line = reader.readLine()) != null) { + splitter.setString(line); + String speed = splitter.next(); + String time = splitter.next(); + Assert.assertTrue(TextUtils.isDigitsOnly(speed)); + Assert.assertTrue(TextUtils.isDigitsOnly(time)); + } + } + } + + PowerProfile powerProfile = PowerProfile.init(mContext); + Assert.assertNotNull(powerProfile); + Assert.assertTrue(powerProfile.isSupported()); + + KernelCpuSpeedReader[] readers = new KernelCpuSpeedReader[powerProfile.getNumCpuClusters()]; + int firstCpuOfCluster = 0; + for (int i = 0; i < powerProfile.getNumCpuClusters(); i++) { + final int numSpeedSteps = powerProfile.getNumSpeedStepsInCpuCluster(i); + readers[i] = new KernelCpuSpeedReader(firstCpuOfCluster, numSpeedSteps); + firstCpuOfCluster += powerProfile.getNumCoresInCpuCluster(i); + } + + for (int i = 0; i < readers.length; i++) { + KernelCpuSpeedReader kernelCpuSpeedReader = readers[i]; + Assert.assertNotNull(kernelCpuSpeedReader); + Assert.assertTrue(kernelCpuSpeedReader.readTotoal() > 0); + long[] cpuCoreJiffies = kernelCpuSpeedReader.readAbsolute(); + Assert.assertEquals(powerProfile.getNumSpeedStepsInCpuCluster(i), cpuCoreJiffies.length); + for (int j = 0; j < cpuCoreJiffies.length; j++) { + Assert.assertTrue(cpuCoreJiffies[j] >= 0); + } + } + } + + @Test + public void testConfigureCpuLoad() throws IOException { + PowerProfile powerProfile = PowerProfile.init(mContext); + Assert.assertNotNull(powerProfile); + Assert.assertTrue(powerProfile.isSupported()); + + int cpuCoreNum = BatteryCanaryUtil.getCpuCoreNum(); + int cpuCoreNum2 = 0; + for (int i = 0; i < powerProfile.getNumCpuClusters(); i++) { + cpuCoreNum2 += powerProfile.getNumCoresInCpuCluster(i); + } + Assert.assertEquals(cpuCoreNum2, cpuCoreNum); + + KernelCpuSpeedReader[] readers = new KernelCpuSpeedReader[cpuCoreNum]; + for (int i = 0; i < cpuCoreNum; i++) { + final int numSpeedSteps = powerProfile.getNumSpeedStepsInCpuCluster(powerProfile.getClusterByCpuNum(i)); + readers[i] = new KernelCpuSpeedReader(i, numSpeedSteps); + } + + long cpuJiffiesBgn = 0; + for (int i = 0; i < readers.length; i++) { + KernelCpuSpeedReader kernelCpuSpeedReader = readers[i]; + Assert.assertNotNull(kernelCpuSpeedReader); + long cpuCoreJiffies = kernelCpuSpeedReader.readTotoal(); + Assert.assertTrue("idx = " + i + ", read = " + Arrays.toString(kernelCpuSpeedReader.readAbsolute()), cpuCoreJiffies > 0); + cpuJiffiesBgn += cpuCoreJiffies; + } + long appJiffiesBgn = ProcStatUtil.of(Process.myPid()).getJiffies(); + + CpuConsumption.hanoi(20); + + long cpuJiffiesEnd = 0; + for (int i = 0; i < readers.length; i++) { + KernelCpuSpeedReader kernelCpuSpeedReader = readers[i]; + Assert.assertNotNull(kernelCpuSpeedReader); + long cpuCoreJiffies = kernelCpuSpeedReader.readTotoal(); + Assert.assertTrue(cpuCoreJiffies > 0); + cpuJiffiesEnd += cpuCoreJiffies; + } + long appJiffiesEnd = ProcStatUtil.of(Process.myPid()).getJiffies(); + + long cpuJiffiesDelta = cpuJiffiesEnd - cpuJiffiesBgn; + long appJiffiesDelta = appJiffiesEnd - appJiffiesBgn; + Assert.assertTrue(cpuJiffiesDelta > 0); + Assert.assertTrue(appJiffiesDelta > 0); + Assert.assertTrue(appJiffiesDelta < cpuJiffiesDelta); + + float cpuLoad = (float) appJiffiesDelta / cpuJiffiesDelta; + float cpuLoadAvg = cpuLoad * cpuCoreNum; + + if (!TestUtils.isAssembleTest()) { + Assert.fail("CPU Load: " + (int) (cpuLoadAvg * 100) + "%"); + } + } + + @Test + public void testConfigureCpuLoadWithMultiThread() throws IOException { + PowerProfile powerProfile = PowerProfile.init(mContext); + Assert.assertNotNull(powerProfile); + Assert.assertTrue(powerProfile.isSupported()); + + int cpuCoreNum = BatteryCanaryUtil.getCpuCoreNum(); + int cpuCoreNum2 = 0; + for (int i = 0; i < powerProfile.getNumCpuClusters(); i++) { + cpuCoreNum2 += powerProfile.getNumCoresInCpuCluster(i); + } + Assert.assertEquals(cpuCoreNum2, cpuCoreNum); + + KernelCpuSpeedReader[] readers = new KernelCpuSpeedReader[cpuCoreNum]; + for (int i = 0; i < cpuCoreNum; i++) { + final int numSpeedSteps = powerProfile.getNumSpeedStepsInCpuCluster(powerProfile.getClusterByCpuNum(i)); + readers[i] = new KernelCpuSpeedReader(i, numSpeedSteps); + } + + long cpuJiffiesBgn = 0; + for (int i = 0; i < readers.length; i++) { + KernelCpuSpeedReader kernelCpuSpeedReader = readers[i]; + Assert.assertNotNull(kernelCpuSpeedReader); + long cpuCoreJiffies = kernelCpuSpeedReader.readTotoal(); + Assert.assertTrue("idx = " + i + ", read = " + Arrays.toString(kernelCpuSpeedReader.readAbsolute()), cpuCoreJiffies > 0); + cpuJiffiesBgn += cpuCoreJiffies; + } + long appJiffiesBgn = ProcStatUtil.of(Process.myPid()).getJiffies(); + + for (int i = 0; i < cpuCoreNum - 1; i++) { + new Thread(new Runnable() { + @Override + public void run() { + while (true) {} + } + }).start(); + } + CpuConsumption.hanoi(20); + + long cpuJiffiesEnd = 0; + for (int i = 0; i < readers.length; i++) { + KernelCpuSpeedReader kernelCpuSpeedReader = readers[i]; + Assert.assertNotNull(kernelCpuSpeedReader); + long cpuCoreJiffies = kernelCpuSpeedReader.readTotoal(); + Assert.assertTrue(cpuCoreJiffies > 0); + cpuJiffiesEnd += cpuCoreJiffies; + } + long appJiffiesEnd = ProcStatUtil.of(Process.myPid()).getJiffies(); + + long cpuJiffiesDelta = cpuJiffiesEnd - cpuJiffiesBgn; + long appJiffiesDelta = appJiffiesEnd - appJiffiesBgn; + Assert.assertTrue(cpuJiffiesDelta > 0); + Assert.assertTrue(appJiffiesDelta > 0); + Assert.assertTrue(appJiffiesDelta < cpuJiffiesDelta); + + float cpuLoad = (float) appJiffiesDelta / cpuJiffiesDelta; + float cpuLoadAvg = cpuLoad * cpuCoreNum; + + if (!TestUtils.isAssembleTest()) { + Assert.fail("CPU Load: " + (int) (cpuLoadAvg * 100) + "%"); + } + } + + @Test + public void testGetCpuPowers() throws IOException { + PowerProfile powerProfile = PowerProfile.init(mContext); + Assert.assertNotNull(powerProfile); + Assert.assertTrue(powerProfile.isSupported()); + + double[] clusterPowers = new double[powerProfile.getNumCpuClusters()]; + for (int i = 0; i < powerProfile.getNumCpuClusters(); i++) { + double clusterPower = powerProfile.getAveragePowerForCpuCluster(i); + clusterPowers[i] = clusterPower; + } + + List cpuCoreStepPowers = new ArrayList<>(); + for (int i = 0; i < powerProfile.getCpuCoreNum(); i++) { + int cluster = powerProfile.getClusterByCpuNum(i); + int steps = powerProfile.getNumSpeedStepsInCpuCluster(cluster); + double[] stepPowers = new double[steps]; + for (int step = 0; step < steps; step++) { + stepPowers[step] = powerProfile.getAveragePowerForCpuCore(cluster, step); + } + cpuCoreStepPowers.add(stepPowers); + } + + for (double[] stepPowers : cpuCoreStepPowers) { + for (double item : stepPowers) { + Assert.assertTrue(item > 0); + } + } + } + + @Test + public void testConfigureCpuBatterySipping() throws IOException { + PowerProfile powerProfile = PowerProfile.init(mContext); + Assert.assertNotNull(powerProfile); + Assert.assertTrue(powerProfile.isSupported()); + + int cpuCoreNum = powerProfile.getCpuCoreNum(); + List cpuCoreStepPowers = new ArrayList<>(); + for (int i = 0; i < cpuCoreNum; i++) { + int cluster = powerProfile.getClusterByCpuNum(i); + int steps = powerProfile.getNumSpeedStepsInCpuCluster(cluster); + double[] stepPowers = new double[steps]; + for (int step = 0; step < steps; step++) { + stepPowers[step] = powerProfile.getAveragePowerForCpuCore(cluster, step); + } + cpuCoreStepPowers.add(stepPowers); + } + + for (double[] stepPowers : cpuCoreStepPowers) { + for (double item : stepPowers) { + Assert.assertTrue(item > 0); + } + } + + KernelCpuSpeedReader[] readers = new KernelCpuSpeedReader[cpuCoreNum]; + for (int i = 0; i < cpuCoreNum; i++) { + final int numSpeedSteps = powerProfile.getNumSpeedStepsInCpuCluster(powerProfile.getClusterByCpuNum(i)); + readers[i] = new KernelCpuSpeedReader(i, numSpeedSteps); + } + + long jiffiesSum = 0; + List cpuCoreStepSips = new ArrayList<>(); + Assert.assertEquals(cpuCoreStepPowers.size(), readers.length); + for (int i = 0; i < readers.length; i++) { + double[] stepPowers = cpuCoreStepPowers.get(i); + KernelCpuSpeedReader reader = readers[i]; + long[] stepJiffies = reader.readAbsolute(); + Assert.assertEquals(stepPowers.length, stepJiffies.length); + double[] stepSips = new double[stepJiffies.length]; + for (int j = 0; j < stepJiffies.length; j++) { + long jiffies = stepJiffies[j]; + jiffiesSum += jiffies; + double power = stepPowers[j]; + double sip = power * ((double) jiffies * JIFFY_MILLIS / ONE_HOR); + stepSips[j] = sip; + } + cpuCoreStepSips.add(stepSips); + } + + double sipSum = 0; + for (double[] stepSips : cpuCoreStepSips) { + for (double item : stepSips) { + Assert.assertTrue(item >= 0); + sipSum += item; + } + } + Assert.assertTrue(sipSum >= 0); + if (!TestUtils.isAssembleTest()) { + Assert.fail("CPU JiffyHour: " + ((float) (jiffiesSum) * JIFFY_MILLIS) / ONE_HOR + + "\nCPU Sipping: " + sipSum + " mAh"); + } + } + + @Test + public void testConfigureCpuBatterySippingDelta() throws IOException { + PowerProfile powerProfile = PowerProfile.init(mContext); + Assert.assertNotNull(powerProfile); + Assert.assertTrue(powerProfile.isSupported()); + + int cpuCoreNum = powerProfile.getCpuCoreNum(); + List cpuCoreStepPowers = new ArrayList<>(); + for (int i = 0; i < cpuCoreNum; i++) { + int cluster = powerProfile.getClusterByCpuNum(i); + int steps = powerProfile.getNumSpeedStepsInCpuCluster(cluster); + double[] stepPowers = new double[steps]; + for (int step = 0; step < steps; step++) { + stepPowers[step] = powerProfile.getAveragePowerForCpuCore(cluster, step); + } + cpuCoreStepPowers.add(stepPowers); + } + + for (double[] stepPowers : cpuCoreStepPowers) { + for (double item : stepPowers) { + Assert.assertTrue(item > 0); + } + } + + KernelCpuSpeedReader[] readers = new KernelCpuSpeedReader[cpuCoreNum]; + for (int i = 0; i < cpuCoreNum; i++) { + final int numSpeedSteps = powerProfile.getNumSpeedStepsInCpuCluster(powerProfile.getClusterByCpuNum(i)); + readers[i] = new KernelCpuSpeedReader(i, numSpeedSteps); + } + + long cpuJiffiesBgn = 0; + for (int i = 0; i < readers.length; i++) { + KernelCpuSpeedReader kernelCpuSpeedReader = readers[i]; + Assert.assertNotNull(kernelCpuSpeedReader); + long cpuCoreJiffies = kernelCpuSpeedReader.readTotoal(); + Assert.assertTrue("idx = " + i + ", read = " + Arrays.toString(kernelCpuSpeedReader.readAbsolute()), cpuCoreJiffies > 0); + cpuJiffiesBgn += cpuCoreJiffies; + } + long appJiffiesBgn = ProcStatUtil.of(Process.myPid()).getJiffies(); + + List cpuCoreStepJiffies = new ArrayList<>(); + for (int i = 0; i < readers.length; i++) { + KernelCpuSpeedReader reader = readers[i]; + long[] stepJiffies = reader.readAbsolute(); + cpuCoreStepJiffies.add(stepJiffies); + } + + CpuConsumption.hanoi(20); + + long cpuJiffiesEnd = 0; + for (int i = 0; i < readers.length; i++) { + KernelCpuSpeedReader kernelCpuSpeedReader = readers[i]; + Assert.assertNotNull(kernelCpuSpeedReader); + long cpuCoreJiffies = kernelCpuSpeedReader.readTotoal(); + Assert.assertTrue(cpuCoreJiffies > 0); + cpuJiffiesEnd += cpuCoreJiffies; + } + Assert.assertTrue(cpuJiffiesEnd > cpuJiffiesBgn); + + List cpuCoreStepJiffiesDeltas = new ArrayList<>(); + for (int i = 0; i < readers.length; i++) { + long[] stepJiffiesBgns = cpuCoreStepJiffies.get(i); + KernelCpuSpeedReader reader = readers[i]; + long[] stepJiffies = reader.readAbsolute(); + Assert.assertEquals(stepJiffiesBgns.length, stepJiffies.length); + long[] stepJiffiesDeltas = new long[stepJiffies.length]; + for (int step = 0; step < stepJiffies.length; step++) { + long jiffiesBgn = stepJiffiesBgns[step]; + long jiffiesEnd = stepJiffies[step]; + stepJiffiesDeltas[step] = jiffiesEnd - jiffiesBgn; + } + cpuCoreStepJiffiesDeltas.add(stepJiffiesDeltas); + } + + Assert.assertEquals(cpuCoreStepJiffiesDeltas.size(), readers.length); + + List cpuCoreStepSips = new ArrayList<>(); + for (int i = 0; i < readers.length; i++) { + double[] stepPowers = cpuCoreStepPowers.get(i); + long[] stepJiffies = cpuCoreStepJiffiesDeltas.get(i); + Assert.assertEquals(stepPowers.length, stepJiffies.length); + double[] stepSips = new double[stepJiffies.length]; + for (int j = 0; j < stepJiffies.length; j++) { + long jiffies = stepJiffies[j]; + double power = stepPowers[j]; + double sip = power * ((double) jiffies * JIFFY_MILLIS / ONE_HOR); + stepSips[j] = sip; + } + cpuCoreStepSips.add(stepSips); + } + + double sipSum = 0; + for (double[] stepSips : cpuCoreStepSips) { + for (double item : stepSips) { + Assert.assertTrue(item >= 0); + sipSum += item; + } + } + Assert.assertTrue(sipSum >= 0); + if (!TestUtils.isAssembleTest()) { + Assert.fail("CPU JiffyHour: " + ((float) (cpuJiffiesEnd - cpuJiffiesBgn) * JIFFY_MILLIS) / ONE_HOR + + "\nCPU Sipping: " + sipSum + " mAh"); + } + } + + @Test + public void testReadKernelPidCpuSpeedState() throws IOException { + PowerProfile powerProfile = PowerProfile.init(mContext); + Assert.assertNotNull(powerProfile); + Assert.assertTrue(powerProfile.isSupported()); + + List clusterStepJiffies = new ArrayList<>(); + String path = "/proc/" + Process.myPid() + "/time_in_state"; + try (BufferedReader reader = new BufferedReader(new FileReader(new File(path)))) { + TextUtils.SimpleStringSplitter splitter = new TextUtils.SimpleStringSplitter(' '); + String line; + int cluster = -1; + int step = -1; + long[] stepJiffies = null; + while ((line = reader.readLine()) != null) { + if (line.startsWith("cpu")) { + cluster++; + step = -1; + if (stepJiffies != null) { + clusterStepJiffies.add(stepJiffies); + } + int stepNum = powerProfile.getNumSpeedStepsInCpuCluster(cluster); + stepJiffies = new long[stepNum]; + continue; + } + step++; + splitter.setString(line); + String speed = splitter.next(); + String time = splitter.next(); + Assert.assertTrue(TextUtils.isDigitsOnly(speed)); + Assert.assertTrue(TextUtils.isDigitsOnly(time)); + stepJiffies[step] = Long.parseLong(time); + } + if (stepJiffies != null) { + clusterStepJiffies.add(stepJiffies); + } + } + + Assert.assertEquals(powerProfile.getNumCpuClusters(), clusterStepJiffies.size()); + for (int i = 0; i < clusterStepJiffies.size(); i++) { + long[] stepJiffies = clusterStepJiffies.get(i); + Assert.assertEquals(powerProfile.getNumSpeedStepsInCpuCluster(i), stepJiffies.length); + } + } + + @Test + public void testReadKernelPidCpuSpeedStateAndProcStatJiffiesCompare() throws IOException, InterruptedException { + PowerProfile powerProfile = PowerProfile.init(mContext); + Assert.assertNotNull(powerProfile); + Assert.assertTrue(powerProfile.isSupported()); + + long kernelPidJiffiesBgn = readKernelPidJiffies(Process.myPid()); + ProcStatUtil.ProcStat procStat = ProcStatUtil.of(Process.myPid()); + + for (int i = 0; i < powerProfile.getCpuCoreNum() - 1; i++) { + new Thread(new Runnable() { + @Override + public void run() { + while (true) {} + } + }).start(); + } + CpuConsumption.hanoi(20); + + long kernelPidJiffiesEnd = readKernelPidJiffies(Process.myPid()); + long procStatJiffies = ProcStatUtil.of(Process.myPid()).getJiffies() - procStat.getJiffies(); + + Assert.assertEquals(procStatJiffies, kernelPidJiffiesEnd - kernelPidJiffiesBgn); + } + + private static long readKernelPidJiffies(int pid) throws IOException { + long kernelPidJiffies = 0; + String path = "/proc/" + pid + "/time_in_state"; + try (BufferedReader reader = new BufferedReader(new FileReader(new File(path)))) { + TextUtils.SimpleStringSplitter splitter = new TextUtils.SimpleStringSplitter(' '); + String line; + while ((line = reader.readLine()) != null) { + if (line.startsWith("cpu")) { + continue; + } + splitter.setString(line); + String speed = splitter.next(); + String time = splitter.next(); + Assert.assertTrue(TextUtils.isDigitsOnly(speed)); + Assert.assertTrue(TextUtils.isDigitsOnly(time)); + kernelPidJiffies += Long.parseLong(time); + } + } + return kernelPidJiffies; + } + + @Test + public void testReadKernelPidCpuSpeedStateListener() throws IOException, InterruptedException { + long start = System.currentTimeMillis(); + for (int i = 0; i < BatteryCanaryUtil.getCpuCoreNum() - 1; i++) { + new Thread(new Runnable() { + @Override + public void run() { + while (true) {} + } + }).start(); + } + + final AtomicBoolean hasCallback = new AtomicBoolean(); + String path = "/proc/" + Process.myPid() + "/time_in_state"; + new FileObserver(path) { + @Override + public void onEvent(int event, @Nullable String path) { + synchronized (hasCallback) { + hasCallback.notifyAll(); + } + } + }.startWatching(); + synchronized (hasCallback) { + hasCallback.wait(); + } + Assert.fail("Time: " + (System.currentTimeMillis() - start)); + } + + @Test + public void testConfigureProcCpuBatterySipping() throws IOException { + PowerProfile powerProfile = PowerProfile.init(mContext); + Assert.assertNotNull(powerProfile); + Assert.assertTrue(powerProfile.isSupported()); + + CpuConsumption.hanoi(20); + ProcStatUtil.ProcStat procStat = ProcStatUtil.of(Process.myPid()); + int[] clusterSteps = new int[powerProfile.getNumCpuClusters()]; + for (int i = 0; i < clusterSteps.length; i++) { + clusterSteps[i] = powerProfile.getNumSpeedStepsInCpuCluster(i); + } + KernelCpuUidFreqTimeReader reader = new KernelCpuUidFreqTimeReader(Process.myPid(), clusterSteps); + List cpuCoreStepJiffies = reader.readAbsolute(); + long jiffySum = 0; + for (long[] stepJiffies : cpuCoreStepJiffies) { + for (long item : stepJiffies) { + jiffySum += item; + } + } + double sipSum = 0; + double figuredJiffiesSum = 0; + for (int i = 0; i < cpuCoreStepJiffies.size(); i++) { + long[] stepJiffies = cpuCoreStepJiffies.get(i); + for (int j = 0; j < stepJiffies.length; j++) { + long jiffy = stepJiffies[j]; + double figuredJiffies = ((double) jiffy / jiffySum) * procStat.getJiffies(); + double power = powerProfile.getAveragePowerForCpuCore(i, j); + double sip = power * (figuredJiffies * JIFFY_MILLIS / ONE_HOR); + sipSum += sip; + figuredJiffiesSum += figuredJiffies; + } + } + + Assert.assertTrue(sipSum >= 0); + Assert.assertEquals(procStat.getJiffies(), figuredJiffiesSum, 1); + + if (!TestUtils.isAssembleTest()) { + Assert.fail("Proc JiffyHour: " + ((float) (procStat.getJiffies()) * JIFFY_MILLIS) / ONE_HOR + + "\nProc Sipping: " + sipSum + " mAh"); + } + } + + + @SuppressWarnings("SpellCheckingInspection") + @Test + public void testJiffiesConsumption() { + long wallTimeBgn = System.currentTimeMillis(); + long upTimeBgn = SystemClock.uptimeMillis(); + long threadTimeBgn = SystemClock.currentThreadTimeMillis(); + + ProcStatUtil.ProcStat procStatBgn = ProcStatUtil.of(Process.myPid(), Process.myTid()); + long threadJiffiesBgn = procStatBgn.getJiffies(); + long threadUtimeBgn = procStatBgn.utime; + long threadStimeBgn = procStatBgn.stime; + long threadCutimeBgn = procStatBgn.cutime; + long threadCstimeBgn = procStatBgn.cstime; + + CpuConsumption.inc(Integer.MAX_VALUE); + + ProcStatUtil.ProcStat procStatEnd = ProcStatUtil.of(Process.myPid(), Process.myTid()); + StringBuilder sb = new StringBuilder().append("wallTimeDiff = ").append(System.currentTimeMillis() - wallTimeBgn) + .append("\nupTimeDiff = ").append(SystemClock.uptimeMillis() - upTimeBgn) + .append("\nthreadTimeDiff = ").append(SystemClock.currentThreadTimeMillis() - threadTimeBgn) + .append("\nthreadJiffiesDiff = ").append(procStatEnd.getJiffies() - threadJiffiesBgn) + .append("\nthreadUtimeDiff = ").append(procStatEnd.utime - threadUtimeBgn) + .append("\nthreadStimeDiff = ").append(procStatEnd.stime - threadStimeBgn) + .append("\nthreadCutimeDiff = ").append(procStatEnd.cutime - threadCutimeBgn) + .append("\nthreadCstimeDiff = ").append(procStatEnd.cstime - threadCstimeBgn) + .append("\n"); + if (!TestUtils.isAssembleTest()) { + Assert.fail(sb.toString()); + } + } + + @SuppressWarnings("SpellCheckingInspection") + @Test + public void testJiffiesConsumption2() { + long wallTimeBgn = System.currentTimeMillis(); + long upTimeBgn = SystemClock.uptimeMillis(); + long threadTimeBgn = SystemClock.currentThreadTimeMillis(); + + ProcStatUtil.ProcStat procStatBgn = ProcStatUtil.of(Process.myPid(), Process.myTid()); + long threadJiffiesBgn = procStatBgn.getJiffies(); + long threadUtimeBgn = procStatBgn.utime; + long threadStimeBgn = procStatBgn.stime; + long threadCutimeBgn = procStatBgn.cutime; + long threadCstimeBgn = procStatBgn.cstime; + + CpuConsumption.hanoi(20); + + ProcStatUtil.ProcStat procStatEnd = ProcStatUtil.of(Process.myPid(), Process.myTid()); + StringBuilder sb = new StringBuilder().append("wallTimeDiff = ").append(System.currentTimeMillis() - wallTimeBgn) + .append("\nupTimeDiff = ").append(SystemClock.uptimeMillis() - upTimeBgn) + .append("\nthreadTimeDiff = ").append(SystemClock.currentThreadTimeMillis() - threadTimeBgn) + .append("\nthreadJiffiesDiff = ").append(procStatEnd.getJiffies() - threadJiffiesBgn) + .append("\nthreadUtimeDiff = ").append(procStatEnd.utime - threadUtimeBgn) + .append("\nthreadStimeDiff = ").append(procStatEnd.stime - threadStimeBgn) + .append("\nthreadCutimeDiff = ").append(procStatEnd.cutime - threadCutimeBgn) + .append("\nthreadCstimeDiff = ").append(procStatEnd.cstime - threadCstimeBgn) + .append("\n"); + if (!TestUtils.isAssembleTest()) { + Assert.fail(sb.toString()); + } + } + + public static class CpuConsumption { + public static void inc(int num) { + for (int i = 0; i < num; i++) { + } + } + + public static void hanoi(int num) { + Stack torre1 = new Stack<>(); + Stack torre2 = new Stack<>(); + Stack torre3 = new Stack<>(); + + for (int i = num; i >= 1; i--) { + torre1.push(i); + } + ordena(torre1.size(), torre1, torre3, torre2); + } + + private static void ordena(int n, Stack torre1, Stack torre3, Stack torre2) { + if (n > 0) { + ordena(n - 1, torre1, torre2, torre3); + torre3.push(torre1.pop()); + Log.i(TAG, "Torre1: " + torre1 + " Torre 2: " + torre2 + " Torre3: " + torre3); + ordena(n - 1, torre2, torre3, torre1); + } + } + } +} diff --git a/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/CanaryUtilsTest.java b/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/CanaryUtilsTest.java index 2e9fbb4e6..e5f155cfa 100644 --- a/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/CanaryUtilsTest.java +++ b/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/CanaryUtilsTest.java @@ -17,6 +17,7 @@ package com.tencent.matrix.batterycanary.utils; import android.app.ActivityManager; +import android.app.ActivityManager.RunningAppProcessInfo; import android.app.Application; import android.app.Notification; import android.app.PendingIntent; @@ -47,6 +48,8 @@ import org.junit.Test; import org.junit.runner.RunWith; +import java.lang.reflect.InvocationTargetException; +import java.util.LinkedList; import java.util.List; import java.util.concurrent.atomic.AtomicBoolean; @@ -302,6 +305,29 @@ private static boolean diceWithBase(int base) { return false; } + @Test + public void testReadAppForegroundStat() throws NoSuchMethodException, InvocationTargetException, IllegalAccessException, NoSuchFieldException { + ActivityManager am = (ActivityManager) mContext.getSystemService(Context.ACTIVITY_SERVICE); + if (am == null) { + return; + } + List runningAppProcesses = am.getRunningAppProcesses(); + if (runningAppProcesses == null) { + return; + } + List myRunningApp = new LinkedList<>(); + for (RunningAppProcessInfo item : runningAppProcesses) { + if (!TextUtils.isEmpty(item.processName) && item.processName.startsWith(mContext.getPackageName())) { + myRunningApp.add(item); + } + } + + Assert.assertEquals(1, myRunningApp.size()); + Assert.assertTrue(myRunningApp.get(0).importance >= RunningAppProcessInfo.IMPORTANCE_FOREGROUND_SERVICE); // 125 + int procState = (int) myRunningApp.get(0).getClass().getDeclaredField("processState").get(myRunningApp.get(0)); + Assert.assertTrue(procState >= 4); // ActivityManager#PROCESS_STATE_BOUND_FOREGROUND_SERVICE, PROCESS_STATE_IMPORTANT_FOREGROUND + } + public static class SpyService extends Service { @Override public IBinder onBind(Intent intent) { diff --git a/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/ProcStatUtilsTest.java b/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/ProcStatUtilsTest.java index 3be9e279b..47047667b 100644 --- a/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/ProcStatUtilsTest.java +++ b/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/ProcStatUtilsTest.java @@ -24,8 +24,12 @@ import android.os.Process; import android.os.SystemClock; import android.text.TextUtils; +import android.util.Pair; +import android.util.SparseArray; import com.tencent.matrix.batterycanary.TestUtils; +import com.tencent.matrix.batterycanary.monitor.feature.JiffiesMonitorFeature; +import com.tencent.matrix.batterycanary.monitor.feature.MonitorFeature; import com.tencent.matrix.util.MatrixLog; import org.junit.After; @@ -953,5 +957,45 @@ public void testParsingIndividualCases() throws IOException, ProcStatUtil.ParseE Assert.assertEquals(exceptedStat.comm, computedStat.comm); } } + + @Test + public void testThreadPollingJiffies() throws InterruptedException { + if (TestUtils.isAssembleTest()) { return; } + + final SparseArray threadProcStatDict = new SparseArray<>(); + int count = 3; + for (int i = 0; i < count; i++) { + Thread thread = new Thread(new Runnable() { + @Override + public void run() { + ProcStatUtil.ProcStat procStat = ProcStatUtil.of(Process.myPid(), Process.myTid()); + threadProcStatDict.put(Process.myTid(), procStat); + while (true) { + try { + Thread.sleep(10L); + } catch (InterruptedException ignored) { + } + } + } + }); + thread.setName("test-jiffies-thread-" + i); + thread.start(); + } + + long runingMillis = 6 * 1000L; + Thread.sleep(runingMillis); + + List jffiesConsumptions = new LinkedList<>(); + for (int i = 0; i < threadProcStatDict.size(); i++) { + ProcStatUtil.ProcStat end = ProcStatUtil.of(Process.myPid(), threadProcStatDict.keyAt(i)); + jffiesConsumptions.add(end.getJiffies() - threadProcStatDict.valueAt(i).getJiffies()); + } + + Assert.assertFalse(jffiesConsumptions.isEmpty()); + for (long item : jffiesConsumptions) { + float cpuLoad = (item * 10f) / runingMillis; + Assert.assertTrue("Thread CPU LOAD: " + cpuLoad, cpuLoad <= 0.005); + } + } } } diff --git a/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/TemperatureUtilsTest.java b/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/TemperatureUtilsTest.java index eec7924cc..f0770112b 100644 --- a/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/TemperatureUtilsTest.java +++ b/matrix/matrix-android/matrix-battery-canary/src/androidTest/java/com/tencent/matrix/batterycanary/utils/TemperatureUtilsTest.java @@ -26,6 +26,8 @@ import android.os.BatteryManager; import androidx.test.platform.app.InstrumentationRegistry; import androidx.test.ext.junit.runners.AndroidJUnit4; + +import android.os.HardwarePropertiesManager; import android.util.Log; import com.tencent.matrix.batterycanary.TestUtils; @@ -67,6 +69,15 @@ public void setUp() { public void shutDown() { } + @Test + public void testGetCpuTemp() { + HardwarePropertiesManager manager = (HardwarePropertiesManager) mContext.getSystemService(Context.HARDWARE_PROPERTIES_SERVICE); + try { + manager.getDeviceTemperatures(HardwarePropertiesManager.DEVICE_TEMPERATURE_CPU, HardwarePropertiesManager.TEMPERATURE_CURRENT); + Assert.fail("Should be permission denied"); + } catch (SecurityException ignored) { + } + } @Test public void testGetDeviceTemperature() throws InterruptedException { diff --git a/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/BatteryMonitorCallback.java b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/BatteryMonitorCallback.java index 200a05882..ee58efa10 100644 --- a/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/BatteryMonitorCallback.java +++ b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/BatteryMonitorCallback.java @@ -7,13 +7,15 @@ import android.text.TextUtils; import android.util.LongSparseArray; -import com.tencent.matrix.Matrix; import com.tencent.matrix.batterycanary.monitor.feature.AbsTaskMonitorFeature.TaskJiffiesSnapshot; import com.tencent.matrix.batterycanary.monitor.feature.AlarmMonitorFeature; import com.tencent.matrix.batterycanary.monitor.feature.AlarmMonitorFeature.AlarmSnapshot; import com.tencent.matrix.batterycanary.monitor.feature.AppStatMonitorFeature; import com.tencent.matrix.batterycanary.monitor.feature.BlueToothMonitorFeature; import com.tencent.matrix.batterycanary.monitor.feature.BlueToothMonitorFeature.BlueToothSnapshot; +import com.tencent.matrix.batterycanary.monitor.feature.CompositeMonitors; +import com.tencent.matrix.batterycanary.monitor.feature.CpuStatFeature; +import com.tencent.matrix.batterycanary.monitor.feature.CpuStatFeature.CpuStateSnapshot; import com.tencent.matrix.batterycanary.monitor.feature.DeviceStatMonitorFeature; import com.tencent.matrix.batterycanary.monitor.feature.DeviceStatMonitorFeature.BatteryTmpSnapshot; import com.tencent.matrix.batterycanary.monitor.feature.DeviceStatMonitorFeature.CpuFreqSnapshot; @@ -25,6 +27,7 @@ import com.tencent.matrix.batterycanary.monitor.feature.LooperTaskMonitorFeature; import com.tencent.matrix.batterycanary.monitor.feature.MonitorFeature.Snapshot.Delta; import com.tencent.matrix.batterycanary.monitor.feature.MonitorFeature.Snapshot.Entry.BeanEntry; +import com.tencent.matrix.batterycanary.monitor.feature.MonitorFeature.Snapshot.Entry.DigitEntry; import com.tencent.matrix.batterycanary.monitor.feature.MonitorFeature.Snapshot.Entry.ListEntry; import com.tencent.matrix.batterycanary.monitor.feature.NotificationMonitorFeature; import com.tencent.matrix.batterycanary.monitor.feature.NotificationMonitorFeature.BadNotification; @@ -37,10 +40,12 @@ import com.tencent.matrix.batterycanary.monitor.feature.WifiMonitorFeature.WifiSnapshot; import com.tencent.matrix.batterycanary.utils.BatteryCanaryUtil; import com.tencent.matrix.batterycanary.utils.Consumer; +import com.tencent.matrix.batterycanary.utils.PowerProfile; import com.tencent.matrix.util.MatrixLog; import java.util.Arrays; import java.util.List; +import java.util.Locale; import java.util.Map; import androidx.annotation.CallSuper; @@ -67,57 +72,65 @@ class BatteryPrinter implements BatteryMonitorCallback { private static final int ONE_MIN = 60 * 1000; @NonNull - private BatteryMonitorCore mMonitor; - private final Printer mPrinter = new Printer(); + protected BatteryMonitorCore mMonitor; + @NonNull + protected CompositeMonitors mCompositeMonitors; + protected final Printer mPrinter = new Printer(); - private long mTraceBgnMillis; - private boolean mIsForeground; + protected long mTraceBgnMillis; + protected boolean mIsForeground; @Nullable AppStats mAppStats; - private final LongSparseArray> tasks = new LongSparseArray<>(); + protected final LongSparseArray> tasks = new LongSparseArray<>(); - @Nullable + // TODO: Remove deprecated fields + @Deprecated protected AlarmMonitorFeature mAlarmFeat; - @Nullable + @Deprecated protected AppStatMonitorFeature mAppStatFeat; - @Nullable + @Deprecated protected BlueToothMonitorFeature mBlueToothFeat; - @Nullable + @Deprecated protected DeviceStatMonitorFeature mDevStatFeat; - @Nullable + @Deprecated protected JiffiesMonitorFeature mJiffiesFeat; - @Nullable + @Deprecated protected LocationMonitorFeature mLocationFeat; - @Nullable + @Deprecated protected TrafficMonitorFeature mTrafficFeat; - @Nullable + @Deprecated protected WakeLockMonitorFeature mWakeLockFeat; - @Nullable + @Deprecated protected WifiMonitorFeature mWifiMonitorFeat; + @Deprecated + protected CpuStatFeature mCpuStatFeat; - @Nullable + @Deprecated protected AlarmSnapshot mLastAlarmSnapshot; - @Nullable + @Deprecated protected BlueToothSnapshot mLastBlueToothSnapshot; - @Nullable + @Deprecated protected BatteryTmpSnapshot mLastBatteryTmpSnapshot; - @Nullable + @Deprecated protected CpuFreqSnapshot mLastCpuFreqSnapshot; - @Nullable + @Deprecated protected JiffiesSnapshot mLastJiffiesSnapshot; - @Nullable + @Deprecated protected LocationSnapshot mLastLocationSnapshot; - @Nullable + @Deprecated protected RadioStatSnapshot mLastTrafficSnapshot; - @Nullable + @Deprecated protected WakeLockSnapshot mLastWakeWakeLockSnapshot; - @Nullable + @Deprecated protected WifiSnapshot mLastWifiSnapshot; + @Deprecated + protected CpuStateSnapshot mLastCpuStateSnapshot; @SuppressWarnings("UnusedReturnValue") @VisibleForTesting - public final BatteryPrinter attach(BatteryMonitorCore monitorCore) { + public BatteryPrinter attach(BatteryMonitorCore monitorCore) { mMonitor = monitorCore; + mCompositeMonitors = new CompositeMonitors(monitorCore); return this; } @@ -141,7 +154,10 @@ protected AppStats getAppStats() { @Override public void onTraceBegin() { mTraceBgnMillis = SystemClock.uptimeMillis(); + mCompositeMonitors.clear(); + mCompositeMonitors.configureAllSnapshot(); + // TODO: Remove deprecated statements // Configure begin snapshots mAlarmFeat = mMonitor.getMonitorFeature(AlarmMonitorFeature.class); if (mAlarmFeat != null) { @@ -185,6 +201,11 @@ public void onTraceBegin() { if (mWifiMonitorFeat != null) { mLastWifiSnapshot = mWifiMonitorFeat.currentSnapshot(); } + + mCpuStatFeat = mMonitor.getMonitorFeature(CpuStatFeature.class); + if (mCpuStatFeat != null && mCpuStatFeat.isSupported()) { + mLastCpuStateSnapshot = mCpuStatFeat.currentCpuStateSnapshot(); + } } @Override @@ -197,6 +218,7 @@ public void onTraceEnd(boolean isForeground) { } mAppStats = AppStats.current(duringMillis).setForeground(isForeground); + mCompositeMonitors.configureDeltas(); onCanaryDump(mAppStats); mAppStats = null; } @@ -340,47 +362,52 @@ protected void onCanaryDump(AppStats appStats) { } @CallSuper - protected void onWritingJiffiesSection(AppStats appStats) { - if (null != mJiffiesFeat && null != mLastJiffiesSnapshot) { - JiffiesSnapshot curr = mJiffiesFeat.currentJiffiesSnapshot(); - Delta delta = curr.diff(mLastJiffiesSnapshot); - - long minute = appStats.getMinute(); - for (ThreadJiffiesEntry threadJiffies : delta.dlt.threadEntries.getList()) { - if (!threadJiffies.stat.toUpperCase().contains("R")) { - continue; - } - // Watching thread state when thread is: - // 1. still running (status 'R') - // 2. runing time > 10min - // 3. avgJiffies > THRESHOLD - long avgJiffies = threadJiffies.get() / minute; - if (appStats.isForeground()) { - if (minute > 10 && avgJiffies > getMonitor().getConfig().fgThreadWatchingLimit) { - MatrixLog.i(TAG, "threadWatchDog fg set, name = " + delta.dlt.name - + ", pid = " + delta.dlt.pid - + ", tid = " + threadJiffies.tid); - mJiffiesFeat.watchBackThreadSate(true, delta.dlt.pid, threadJiffies.tid); - } - } else { - if (minute > 10 && avgJiffies > getMonitor().getConfig().bgThreadWatchingLimit) { - MatrixLog.i(TAG, "threadWatchDog bg set, name = " + delta.dlt.name - + ", pid = " + delta.dlt.pid - + ", tid = " + threadJiffies.tid); - mJiffiesFeat.watchBackThreadSate(false, delta.dlt.pid, threadJiffies.tid); + protected void onWritingJiffiesSection(final AppStats appStats) { + mCompositeMonitors.getDelta(JiffiesSnapshot.class, new Consumer>() { + @Override + public void accept(final Delta delta) { + final long minute = appStats.getMinute(); + for (final ThreadJiffiesEntry threadJiffies : delta.dlt.threadEntries.getList()) { + if (!threadJiffies.stat.toUpperCase().contains("R")) { + continue; } + mCompositeMonitors.getFeature(JiffiesMonitorFeature.class, new Consumer() { + @Override + public void accept(JiffiesMonitorFeature feature) { + // Watching thread state when thread is: + // 1. still running (status 'R') + // 2. runing time > 10min + // 3. avgJiffies > THRESHOLD + long avgJiffies = threadJiffies.get() / minute; + if (appStats.isForeground()) { + if (minute > 10 && avgJiffies > getMonitor().getConfig().fgThreadWatchingLimit) { + MatrixLog.i(TAG, "threadWatchDog fg set, name = " + delta.dlt.name + + ", pid = " + delta.dlt.pid + + ", tid = " + threadJiffies.tid); + feature.watchBackThreadSate(true, delta.dlt.pid, threadJiffies.tid); + } + } else { + if (minute > 10 && avgJiffies > getMonitor().getConfig().bgThreadWatchingLimit) { + MatrixLog.i(TAG, "threadWatchDog bg set, name = " + delta.dlt.name + + ", pid = " + delta.dlt.pid + + ", tid = " + threadJiffies.tid); + feature.watchBackThreadSate(false, delta.dlt.pid, threadJiffies.tid); + } + } + } + }); } + onReportJiffies(delta); + onWritingSectionContent(delta, appStats, mPrinter); } - onReportJiffies(delta); - onWritingSectionContent(delta, appStats, mPrinter); - } + }); } @CallSuper protected void onWritingAppStatSection(final AppStats appStats) { createSection("app_stats", new Consumer() { @Override - public void accept(Printer printer) { + public void accept(final Printer printer) { printer.createSubSection("stat_time"); printer.writeLine("time", appStats.getMinute() + "(min)"); printer.writeLine("fg", String.valueOf(appStats.appFgRatio)); @@ -394,97 +421,114 @@ public void accept(Printer printer) { if (!TextUtils.isEmpty(appStats.sceneTop2)) { printer.writeLine("sceneTop2", appStats.sceneTop2 + "/" + appStats.sceneTop2Ratio); } - - if (mAppStatFeat != null) { - AppStatMonitorFeature.AppStatSnapshot currSnapshot = mAppStatFeat.currentAppStatSnapshot(); - printer.createSubSection("run_time"); - printer.writeLine("time", currSnapshot.uptime.get() / ONE_MIN + "(min)"); - printer.writeLine("fg", String.valueOf(currSnapshot.fgRatio.get())); - printer.writeLine("bg", String.valueOf(currSnapshot.bgRatio.get())); - printer.writeLine("fgSrv", String.valueOf(currSnapshot.fgSrvRatio.get())); - } + mCompositeMonitors.getFeature(AppStatMonitorFeature.class, new Consumer() { + @Override + public void accept(AppStatMonitorFeature feature) { + AppStatMonitorFeature.AppStatSnapshot currSnapshot = feature.currentAppStatSnapshot(); + printer.createSubSection("run_time"); + printer.writeLine("time", currSnapshot.uptime.get() / ONE_MIN + "(min)"); + printer.writeLine("fg", String.valueOf(currSnapshot.fgRatio.get())); + printer.writeLine("bg", String.valueOf(currSnapshot.bgRatio.get())); + printer.writeLine("fgSrv", String.valueOf(currSnapshot.fgSrvRatio.get())); + } + }); } }); } @CallSuper protected void onWritingSections(final AppStats appStats) { - if (/**/(mAlarmFeat != null && mLastAlarmSnapshot != null) - || (mWakeLockFeat != null && mLastWakeWakeLockSnapshot != null) + if (/**/(mCompositeMonitors.getDelta(AlarmSnapshot.class) != null) + || (mCompositeMonitors.getDelta(WakeLockSnapshot.class) != null) ) { // Alarm, WakeLock createSection("awake", new Consumer() { @Override public void accept(Printer printer) { - if (mAlarmFeat != null && mLastAlarmSnapshot != null) { - AlarmSnapshot alarmSnapshot = mAlarmFeat.currentAlarms(); - Delta delta = alarmSnapshot.diff(mLastAlarmSnapshot); - onReportAlarm(delta); - onWritingSectionContent(delta, appStats, mPrinter); - } - if (mWakeLockFeat != null && mLastWakeWakeLockSnapshot != null) { - WakeLockSnapshot wakeLockSnapshot = mWakeLockFeat.currentWakeLocks(); - Delta delta = wakeLockSnapshot.diff(mLastWakeWakeLockSnapshot); - onReportWakeLock(delta); - onWritingSectionContent(delta, appStats, mPrinter); - } + mCompositeMonitors.getDelta(AlarmSnapshot.class, new Consumer>() { + @Override + public void accept(Delta delta) { + onReportAlarm(delta); + onWritingSectionContent(delta, appStats, mPrinter); + } + }); + mCompositeMonitors.getDelta(WakeLockSnapshot.class, new Consumer>() { + @Override + public void accept(Delta delta) { + onReportWakeLock(delta); + onWritingSectionContent(delta, appStats, mPrinter); + } + }); } }); } - if (/**/(mBlueToothFeat != null && mLastBlueToothSnapshot != null) - || (mWifiMonitorFeat != null && mLastWifiSnapshot != null) - || (mLocationFeat != null && mLastLocationSnapshot != null) + if (/**/(mCompositeMonitors.getDelta(BlueToothSnapshot.class) != null) + || (mCompositeMonitors.getDelta(WifiSnapshot.class) != null) + || (mCompositeMonitors.getDelta(LocationSnapshot.class) != null) ) { - // Scanning + // Scanning: BL, WIFI, GPS createSection("scanning", new Consumer() { @Override public void accept(Printer printer) { - if (mBlueToothFeat != null && mLastBlueToothSnapshot != null) { - // BlueTooth - BlueToothSnapshot currSnapshot = mBlueToothFeat.currentSnapshot(); - Delta delta = currSnapshot.diff(mLastBlueToothSnapshot); - onReportBlueTooth(delta); - onWritingSectionContent(delta, appStats, mPrinter); - } - if (mWifiMonitorFeat != null && mLastWifiSnapshot != null) { - // Wifi - WifiSnapshot currSnapshot = mWifiMonitorFeat.currentSnapshot(); - Delta delta = currSnapshot.diff(mLastWifiSnapshot); - onReportWifi(delta); - onWritingSectionContent(delta, appStats, mPrinter); - } - if (mLocationFeat != null && mLastLocationSnapshot != null) { - // Location - LocationSnapshot currSnapshot = mLocationFeat.currentSnapshot(); - Delta delta = currSnapshot.diff(mLastLocationSnapshot); - onReportLocation(delta); - onWritingSectionContent(delta, appStats, mPrinter); - } + // BlueTooth + mCompositeMonitors.getDelta(BlueToothSnapshot.class, new Consumer>() { + @Override + public void accept(Delta delta) { + onReportBlueTooth(delta); + onWritingSectionContent(delta, appStats, mPrinter); + } + }); + // Wifi + mCompositeMonitors.getDelta(WifiSnapshot.class, new Consumer>() { + @Override + public void accept(Delta delta) { + onReportWifi(delta); + onWritingSectionContent(delta, appStats, mPrinter); + } + }); + // Location + mCompositeMonitors.getDelta(LocationSnapshot.class, new Consumer>() { + @Override + public void accept(Delta delta) { + onReportLocation(delta); + onWritingSectionContent(delta, appStats, mPrinter); + } + }); } }); } - if (/**/(mAppStatFeat != null) - || (mDevStatFeat != null && mLastCpuFreqSnapshot != null) - || (mDevStatFeat != null && mLastBatteryTmpSnapshot != null) + if (/**/(mCompositeMonitors.getFeature(AppStatMonitorFeature.class) != null) + || (mCompositeMonitors.getDelta(CpuStateSnapshot.class) != null) + || (mCompositeMonitors.getDelta(CpuFreqSnapshot.class) != null) + || (mCompositeMonitors.getDelta(BatteryTmpSnapshot.class) != null) ) { - // Status + // Stats: Cpu Usage, Device Status createSection("dev_stats", new Consumer() { @Override public void accept(Printer printer) { - if (mDevStatFeat != null && mLastCpuFreqSnapshot != null) { - CpuFreqSnapshot cpuFreqSnapshot = mDevStatFeat.currentCpuFreq(); - final Delta delta = cpuFreqSnapshot.diff(mLastCpuFreqSnapshot); - onReportCpuFreq(delta); - onWritingSectionContent(delta, appStats, mPrinter); - } - if (mDevStatFeat != null && mLastBatteryTmpSnapshot != null) { - BatteryTmpSnapshot batteryTmpSnapshot = mDevStatFeat.currentBatteryTemperature(Matrix.with().getApplication()); - Delta delta = batteryTmpSnapshot.diff(mLastBatteryTmpSnapshot); - onReportTemperature(delta); - onWritingSectionContent(delta, appStats, mPrinter); - } + mCompositeMonitors.getDelta(CpuStateSnapshot.class, new Consumer>() { + @Override + public void accept(Delta delta) { + onReportCpuStats(delta); + onWritingSectionContent(delta, appStats, mPrinter); + } + }); + mCompositeMonitors.getDelta(CpuFreqSnapshot.class, new Consumer>() { + @Override + public void accept(Delta delta) { + onReportCpuFreq(delta); + onWritingSectionContent(delta, appStats, mPrinter); + } + }); + mCompositeMonitors.getDelta(BatteryTmpSnapshot.class, new Consumer>() { + @Override + public void accept(Delta delta) { + onReportTemperature(delta); + onWritingSectionContent(delta, appStats, mPrinter); + } + }); } }); } @@ -498,7 +542,7 @@ protected void onWritingSections() { } @CallSuper - protected boolean onWritingSectionContent(@NonNull Delta sessionDelta, AppStats appStats, Printer printer) { + protected boolean onWritingSectionContent(@NonNull Delta sessionDelta, AppStats appStats, final Printer printer) { // - Dump Jiffies if (sessionDelta.dlt instanceof JiffiesSnapshot) { //noinspection unchecked @@ -617,6 +661,54 @@ protected boolean onWritingSectionContent(@NonNull Delta sessionDelta, AppSta return true; } + // - Dump CpuStats + if (sessionDelta.dlt instanceof CpuStateSnapshot) { + //noinspection unchecked + final Delta delta = (Delta) sessionDelta; + // Cpu Usage + printer.createSubSection("cpu_load"); + printer.writeLine(delta.during + "(mls)\t" + (delta.during / ONE_MIN) + "(min)"); + final CpuStatFeature cpuStatFeature = mCompositeMonitors.getFeature(CpuStatFeature.class); + if (cpuStatFeature != null) { + mCompositeMonitors.getDelta(JiffiesSnapshot.class, new Consumer>() { + @Override + public void accept(Delta jiffiesDelta) { + long appJiffiesDelta = jiffiesDelta.dlt.totalJiffies.get(); + long cpuJiffiesDelta = delta.dlt.totalCpuJiffies(); + float cpuLoad = (float) appJiffiesDelta / cpuJiffiesDelta; + float cpuLoadAvg = cpuLoad * cpuStatFeature.getPowerProfile().getCpuCoreNum(); + printer.writeLine("usage", (int) (cpuLoadAvg * 100) + "%"); + } + }); + } + for (int i = 0; i < delta.dlt.cpuCoreStates.size(); i++) { + ListEntry> listEntry = delta.dlt.cpuCoreStates.get(i); + printer.writeLine("cpu" + i, Arrays.toString(listEntry.getList().toArray())); + } + // BatterySip + if (cpuStatFeature != null) { + // Cpu battery sip - CPU State + final PowerProfile powerProfile = cpuStatFeature.getPowerProfile(); + printer.writeLine("inc_cpu_sip", String.format(Locale.US, "%.2f(mAh)", delta.dlt.configureCpuSip(powerProfile))); + printer.writeLine("cur_cpu_sip", String.format(Locale.US, "%.2f(mAh)", delta.end.configureCpuSip(powerProfile))); + // Cpu battery sip - Proc State + mCompositeMonitors.getDelta(JiffiesSnapshot.class, new Consumer>() { + @Override + public void accept(Delta jiffiesDelta) { + double procSipDelta = delta.dlt.configureProcSip(powerProfile, jiffiesDelta.dlt.totalJiffies.get()); + double procSipEnd = delta.end.configureProcSip(powerProfile, jiffiesDelta.end.totalJiffies.get()); + printer.writeLine("inc_prc_sip", String.format(Locale.US, "%.2f(mAh)", procSipDelta)); + printer.writeLine("cur_prc_sip", String.format(Locale.US, "%.2f(mAh)", procSipEnd)); + if (Double.isNaN(procSipDelta)) { + double procSipBgn = delta.bgn.configureProcSip(powerProfile, jiffiesDelta.bgn.totalJiffies.get()); + printer.writeLine("inc_prc_sipr", String.format(Locale.US, "%.2f(mAh)", procSipEnd - procSipBgn)); + } + } + }); + } + return true; + } + // - Dump Battery Temperature if (sessionDelta.dlt instanceof BatteryTmpSnapshot) { //noinspection unchecked @@ -645,6 +737,9 @@ protected void onReportBlueTooth(@NonNull Delta delta) { protected void onReportCpuFreq(@NonNull Delta delta) { } + protected void onReportCpuStats(@NonNull Delta delta) { + } + protected void onReportJiffies(@NonNull Delta delta) { } diff --git a/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/feature/AbsTaskMonitorFeature.java b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/feature/AbsTaskMonitorFeature.java index e2a204fe3..e5280e9ad 100644 --- a/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/feature/AbsTaskMonitorFeature.java +++ b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/feature/AbsTaskMonitorFeature.java @@ -445,7 +445,7 @@ protected TaskJiffiesSnapshot createSnapshot(String name, int tid) { } if (mCore.getConfig().isUseThreadClock) { - snapshot.jiffies = DigitEntry.of(SystemClock.currentThreadTimeMillis() / 10); + snapshot.jiffies = DigitEntry.of(SystemClock.currentThreadTimeMillis() / BatteryCanaryUtil.JIFFY_MILLIS); } else { int pid = Process.myPid(); ProcStatUtil.ProcStat stat = ProcStatUtil.of(pid, tid); diff --git a/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/feature/CompositeMonitors.java b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/feature/CompositeMonitors.java new file mode 100644 index 000000000..a1a9e9a95 --- /dev/null +++ b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/feature/CompositeMonitors.java @@ -0,0 +1,177 @@ +package com.tencent.matrix.batterycanary.monitor.feature; + +import com.tencent.matrix.batterycanary.monitor.BatteryMonitorCore; +import com.tencent.matrix.batterycanary.monitor.feature.MonitorFeature.Snapshot; +import com.tencent.matrix.batterycanary.monitor.feature.MonitorFeature.Snapshot.Delta; +import com.tencent.matrix.batterycanary.utils.Consumer; + +import java.util.HashMap; +import java.util.Map; + +import androidx.annotation.CallSuper; +import androidx.annotation.Nullable; + +/** + * @author Kaede + * @since 2021/9/18 + */ +public class CompositeMonitors { + protected final Map>, Snapshot> mBgnSnapshots = new HashMap<>(); + protected final Map>, Delta> mDeltas = new HashMap<>(); + + protected final BatteryMonitorCore mMonitor; + + public CompositeMonitors(BatteryMonitorCore core) { + mMonitor = core; + } + + public void clear() { + mBgnSnapshots.clear(); + mDeltas.clear(); + } + + @Nullable + public T getFeature(Class clazz) { + for (MonitorFeature plugin : mMonitor.getConfig().features) { + if (clazz.isAssignableFrom(plugin.getClass())) { + //noinspection unchecked + return (T) plugin; + } + } + return null; + } + + public void getFeature(Class clazz, Consumer block) { + T feature = getFeature(clazz); + if (feature != null) { + block.accept(feature); + } + } + + @Nullable + public > Delta getDelta(Class snapshotClass) { + //noinspection unchecked + return (Delta) mDeltas.get(snapshotClass); + } + + public > void getDelta(Class snapshotClass, Consumer> block) { + Delta delta = getDelta(snapshotClass); + if (delta != null) { + block.accept(delta); + } + } + + @CallSuper + public void configureAllSnapshot() { + statCurrSnapshot(AlarmMonitorFeature.AlarmSnapshot.class); + statCurrSnapshot(BlueToothMonitorFeature.BlueToothSnapshot.class); + statCurrSnapshot(DeviceStatMonitorFeature.CpuFreqSnapshot.class); + statCurrSnapshot(DeviceStatMonitorFeature.BatteryTmpSnapshot.class); + statCurrSnapshot(JiffiesMonitorFeature.JiffiesSnapshot.class); + statCurrSnapshot(LocationMonitorFeature.LocationSnapshot.class); + statCurrSnapshot(TrafficMonitorFeature.RadioStatSnapshot.class); + statCurrSnapshot(WakeLockMonitorFeature.WakeLockSnapshot.class); + statCurrSnapshot(WifiMonitorFeature.WifiSnapshot.class); + statCurrSnapshot(CpuStatFeature.CpuStateSnapshot.class); + } + + @SuppressWarnings({"unchecked", "rawtypes"}) + public void configureDeltas() { + for (Map.Entry>, Snapshot> item : mBgnSnapshots.entrySet()) { + Snapshot lastSnapshot = item.getValue(); + if (lastSnapshot != null) { + Class> snapshotClass = item.getKey(); + Snapshot currSnapshot = statCurrSnapshot(snapshotClass); + if (currSnapshot != null && currSnapshot.getClass() == lastSnapshot.getClass()) { + mDeltas.put(snapshotClass, currSnapshot.diff(lastSnapshot)); + } + } + } + } + + @CallSuper + protected Snapshot statCurrSnapshot(Class> snapshotClass) { + Snapshot snapshot = null; + if (snapshotClass == AlarmMonitorFeature.AlarmSnapshot.class) { + AlarmMonitorFeature feature = mMonitor.getMonitorFeature(AlarmMonitorFeature.class); + if (feature != null) { + snapshot = feature.currentAlarms(); + mBgnSnapshots.put(snapshotClass, snapshot); + } + return snapshot; + } + if (snapshotClass == BlueToothMonitorFeature.BlueToothSnapshot.class) { + BlueToothMonitorFeature feature = mMonitor.getMonitorFeature(BlueToothMonitorFeature.class); + if (feature != null) { + snapshot = feature.currentSnapshot(); + mBgnSnapshots.put(snapshotClass, snapshot); + } + return snapshot; + } + if (snapshotClass == DeviceStatMonitorFeature.CpuFreqSnapshot.class) { + DeviceStatMonitorFeature feature = mMonitor.getMonitorFeature(DeviceStatMonitorFeature.class); + if (feature != null) { + snapshot = feature.currentCpuFreq(); + mBgnSnapshots.put(snapshotClass, snapshot); + } + return snapshot; + } + if (snapshotClass == DeviceStatMonitorFeature.BatteryTmpSnapshot.class) { + DeviceStatMonitorFeature feature = mMonitor.getMonitorFeature(DeviceStatMonitorFeature.class); + if (feature != null) { + snapshot = feature.currentBatteryTemperature(mMonitor.getContext()); + mBgnSnapshots.put(snapshotClass, snapshot); + } + return snapshot; + } + if (snapshotClass == JiffiesMonitorFeature.JiffiesSnapshot.class) { + JiffiesMonitorFeature feature = mMonitor.getMonitorFeature(JiffiesMonitorFeature.class); + if (feature != null) { + snapshot = feature.currentJiffiesSnapshot(); + mBgnSnapshots.put(snapshotClass, snapshot); + } + return snapshot; + } + if (snapshotClass == LocationMonitorFeature.LocationSnapshot.class) { + LocationMonitorFeature feature = mMonitor.getMonitorFeature(LocationMonitorFeature.class); + if (feature != null) { + snapshot = feature.currentSnapshot(); + mBgnSnapshots.put(snapshotClass, snapshot); + } + return snapshot; + } + if (snapshotClass == TrafficMonitorFeature.RadioStatSnapshot.class) { + TrafficMonitorFeature feature = mMonitor.getMonitorFeature(TrafficMonitorFeature.class); + if (feature != null) { + snapshot = feature.currentRadioSnapshot(mMonitor.getContext()); + mBgnSnapshots.put(snapshotClass, snapshot); + } + return snapshot; + } + if (snapshotClass == WakeLockMonitorFeature.WakeLockSnapshot.class) { + WakeLockMonitorFeature feature = mMonitor.getMonitorFeature(WakeLockMonitorFeature.class); + if (feature != null) { + snapshot = feature.currentWakeLocks(); + mBgnSnapshots.put(snapshotClass, snapshot); + } + return snapshot; + } + if (snapshotClass == WifiMonitorFeature.WifiSnapshot.class) { + WifiMonitorFeature feature = mMonitor.getMonitorFeature(WifiMonitorFeature.class); + if (feature != null) { + snapshot = feature.currentSnapshot(); + mBgnSnapshots.put(snapshotClass, snapshot); + } + return snapshot; + } + if (snapshotClass == CpuStatFeature.CpuStateSnapshot.class) { + CpuStatFeature feature = mMonitor.getMonitorFeature(CpuStatFeature.class); + if (feature != null && feature.isSupported()) { + snapshot = feature.currentCpuStateSnapshot(); + mBgnSnapshots.put(snapshotClass, snapshot); + } + return snapshot; + } + return null; + } +} diff --git a/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/feature/CpuStatFeature.java b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/feature/CpuStatFeature.java new file mode 100644 index 000000000..d1af3617d --- /dev/null +++ b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/monitor/feature/CpuStatFeature.java @@ -0,0 +1,237 @@ +package com.tencent.matrix.batterycanary.monitor.feature; + +import android.os.Process; + +import com.tencent.matrix.batterycanary.monitor.feature.MonitorFeature.Snapshot.Entry.DigitEntry; +import com.tencent.matrix.batterycanary.monitor.feature.MonitorFeature.Snapshot.Entry.ListEntry; +import com.tencent.matrix.batterycanary.utils.KernelCpuSpeedReader; +import com.tencent.matrix.batterycanary.utils.KernelCpuUidFreqTimeReader; +import com.tencent.matrix.batterycanary.utils.PowerProfile; +import com.tencent.matrix.util.MatrixLog; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import androidx.annotation.WorkerThread; + +import static com.tencent.matrix.batterycanary.utils.BatteryCanaryUtil.JIFFY_MILLIS; +import static com.tencent.matrix.batterycanary.utils.BatteryCanaryUtil.ONE_HOR; + +/** + * @author Kaede + * @since 2021/9/10 + */ +@SuppressWarnings("SpellCheckingInspection") +public class CpuStatFeature extends AbsTaskMonitorFeature { + private static final String TAG = "Matrix.battery.CpuStatFeature"; + private PowerProfile mPowerProfile; + + @Override + protected String getTag() { + return TAG; + } + + @Override + public int weight() { + return 0; + } + + @Override + public void onTurnOn() { + super.onTurnOn(); + tryInitPowerProfile(); + } + + @Override + public void onForeground(boolean isForeground) { + super.onForeground(isForeground); + if (!isForeground) { + if (mPowerProfile == null) { + mCore.getHandler().post(new Runnable() { + @Override + public void run() { + tryInitPowerProfile(); + } + }); + } + } + } + + @WorkerThread + private void tryInitPowerProfile() { + if (mPowerProfile != null) { + return; + } + synchronized (this) { + if (mPowerProfile != null) { + return; + } + try { + // Check PowerProfile compat + mPowerProfile = PowerProfile.init(mCore.getContext()); + // Check KernelCpuSpeedReader compat + for (int i = 0; i < mPowerProfile.getCpuCoreNum(); i++) { + final int numSpeedSteps = mPowerProfile.getNumSpeedStepsInCpuCluster(mPowerProfile.getClusterByCpuNum(i)); + new KernelCpuSpeedReader(i, numSpeedSteps).smoke(); + } + // Check KernelCpuUidFreqTimeReader compat + int[] clusterSteps = new int[mPowerProfile.getNumCpuClusters()]; + for (int i = 0; i < clusterSteps.length; i++) { + clusterSteps[i] = mPowerProfile.getNumSpeedStepsInCpuCluster(i); + } + new KernelCpuUidFreqTimeReader(Process.myPid(), clusterSteps).smoke(); + } catch (IOException e) { + MatrixLog.w(TAG, "Init cpuStat failed: " + e.getMessage()); + mPowerProfile = null; + } + } + } + + public boolean isSupported() { + return mPowerProfile != null; + } + + public PowerProfile getPowerProfile() { + return mPowerProfile; + } + + public CpuStateSnapshot currentCpuStateSnapshot() { + CpuStateSnapshot snapshot = new CpuStateSnapshot(); + try { + if (!isSupported()) { + throw new IOException("PowerProfile not supported"); + } + synchronized (this) { + if (!isSupported()) { + throw new IOException("PowerProfile not supported"); + } + // Cpu core steps jiffies + snapshot.cpuCoreStates = new ArrayList<>(); + for (int i = 0; i < mPowerProfile.getCpuCoreNum(); i++) { + final int numSpeedSteps = mPowerProfile.getNumSpeedStepsInCpuCluster(mPowerProfile.getClusterByCpuNum(i)); + KernelCpuSpeedReader cpuStepJiffiesReader = new KernelCpuSpeedReader(i, numSpeedSteps); + long[] cpuCoreStepJiffies = cpuStepJiffiesReader.readAbsolute(); + ListEntry> cpuCoreState = ListEntry.ofDigits(cpuCoreStepJiffies); + snapshot.cpuCoreStates.add(cpuCoreState); + } + + // Proc cluster steps jiffies + int[] clusterSteps = new int[mPowerProfile.getNumCpuClusters()]; + for (int i = 0; i < clusterSteps.length; i++) { + clusterSteps[i] = mPowerProfile.getNumSpeedStepsInCpuCluster(i); + } + KernelCpuUidFreqTimeReader procStepJiffiesReader = new KernelCpuUidFreqTimeReader(Process.myPid(), clusterSteps); + List procStepJiffies = procStepJiffiesReader.readAbsolute(); + snapshot.procCpuCoreStates = new ArrayList<>(); + for (long[] item : procStepJiffies) { + ListEntry> procCpuCoreState = ListEntry.ofDigits(item); + snapshot.procCpuCoreStates.add(procCpuCoreState); + } + } + } catch (Exception e) { + MatrixLog.w(TAG, "Read cpu core state fail: " + e.getMessage()); + snapshot.setValid(false); + } + return snapshot; + } + + public static final class CpuStateSnapshot extends Snapshot { + /* + * cpuCoreStates + * [ + * [step1Jiffies, step2Jiffies ...], // CpuCore 1 + * [step1Jiffies, step2Jiffies ...], // CpuCore 2 + * ... + * ] + * + * procCpuCoreStates + * [ + * [step1Jiffies, step2Jiffies ...], // Cluster 1 + * [step1Jiffies, step2Jiffies ...], // Cluster 2 + * ... + * ] + */ + public List>> cpuCoreStates = Collections.emptyList(); + public List>> procCpuCoreStates = Collections.emptyList(); + + public long totalCpuJiffies() { + long sum = 0; + for (ListEntry> cpuCoreState : cpuCoreStates) { + for (DigitEntry item : cpuCoreState.getList()) { + sum += item.value; + } + } + return sum; + } + + public double configureCpuSip(PowerProfile powerProfile) { + if (!powerProfile.isSupported()) { + return 0; + } + double sipSum = 0; + for (int i = 0; i < cpuCoreStates.size(); i++) { + List> stepJiffies = cpuCoreStates.get(i).getList(); + for (int j = 0; j < stepJiffies.size(); j++) { + double jiffy = stepJiffies.get(j).get(); + int cluster = powerProfile.getClusterByCpuNum(i); + double power = powerProfile.getAveragePowerForCpuCore(cluster, j); + double sip = power * (jiffy * JIFFY_MILLIS / ONE_HOR); + sipSum += sip; + } + } + return sipSum; + } + + public double configureProcSip(PowerProfile powerProfile, long procJiffies) { + if (!powerProfile.isSupported()) { + return 0; + } + long jiffySum = 0; + for (ListEntry> stepJiffies : procCpuCoreStates) { + for (DigitEntry item : stepJiffies.getList()) { + jiffySum += item.get(); + } + } + double sipSum = 0; + for (int i = 0; i < procCpuCoreStates.size(); i++) { + List> stepJiffies = procCpuCoreStates.get(i).getList(); + for (int j = 0; j < stepJiffies.size(); j++) { + long jiffy = stepJiffies.get(j).get(); + double figuredJiffies = ((double) jiffy / jiffySum) * procJiffies; + double power = powerProfile.getAveragePowerForCpuCore(i, j); + double sip = power * (figuredJiffies * JIFFY_MILLIS / ONE_HOR); + sipSum += sip; + } + } + return sipSum; + } + + CpuStateSnapshot() { + } + + @Override + public Delta diff(CpuStateSnapshot bgn) { + return new Delta(bgn, this) { + @Override + protected CpuStateSnapshot computeDelta() { + CpuStateSnapshot delta = new CpuStateSnapshot(); + if (bgn.cpuCoreStates.size() != end.cpuCoreStates.size()) { + delta.setValid(false); + } else { + delta.cpuCoreStates = new ArrayList<>(); + for (int i = 0; i < end.cpuCoreStates.size(); i++) { + delta.cpuCoreStates.add(Differ.ListDiffer.globalDiff(bgn.cpuCoreStates.get(i), end.cpuCoreStates.get(i))); + } + delta.procCpuCoreStates = new ArrayList<>(); + for (int i = 0; i < end.procCpuCoreStates.size(); i++) { + delta.procCpuCoreStates.add(Differ.ListDiffer.globalDiff(bgn.procCpuCoreStates.get(i), end.procCpuCoreStates.get(i))); + } + } + return delta; + } + }; + } + } +} diff --git a/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/BatteryCanaryUtil.java b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/BatteryCanaryUtil.java index bd6a8d031..d7110058f 100644 --- a/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/BatteryCanaryUtil.java +++ b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/BatteryCanaryUtil.java @@ -62,7 +62,11 @@ public final class BatteryCanaryUtil { private static final String TAG = "Matrix.battery.Utils"; private static final int DEFAULT_MAX_STACK_LAYER = 10; private static final int DEFAULT_AMS_CACHE_MILLIS = 5 * 1000; - public static final int ONE_MIN = 60 * 1000; + + public static final int ONE_MIN = 60 * 1000; + public static final int ONE_HOR = 60 * 60 * 1000; + public static final int JIFFY_HZ = 100; // @Os.sysconf(OsConstants._SC_CLK_TCK) + public static final int JIFFY_MILLIS = 1000 / JIFFY_HZ; public interface Proxy { String getProcessName(); @@ -251,8 +255,8 @@ public static String getAlarmTypeString(final int type) { } public static int[] getCpuCurrentFreq() { - int[] output = new int[getNumCores()]; - for (int i = 0; i < getNumCores(); i++) { + int[] output = new int[getCpuCoreNum()]; + for (int i = 0; i < getCpuCoreNum(); i++) { output[i] = 0; String path = "/sys/devices/system/cpu/cpu" + i + "/cpufreq/scaling_cur_freq"; String cat = cat(path); @@ -267,7 +271,7 @@ public static int[] getCpuCurrentFreq() { return output; } - private static int getNumCores() { + public static int getCpuCoreNum() { try { // Get directory containing CPU info File dir = new File("/sys/devices/system/cpu/"); diff --git a/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/KernelCpuSpeedReader.java b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/KernelCpuSpeedReader.java new file mode 100644 index 000000000..cd927965a --- /dev/null +++ b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/KernelCpuSpeedReader.java @@ -0,0 +1,71 @@ +package com.tencent.matrix.batterycanary.utils; + + +import android.text.TextUtils; + +import java.io.BufferedReader; +import java.io.FileReader; +import java.io.IOException; + +import androidx.annotation.RestrictTo; + +/** + * Reads CPU time of a specific core spent at various frequencies and provides a delta from the + * last call to {@link #readDelta}. Each line in the proc file has the format: + *

+ * freq time + *

+ * where time is measured in jiffies. + * + * @see com.android.internal.os.KernelCpuSpeedReader + */ +@RestrictTo(RestrictTo.Scope.LIBRARY) +@SuppressWarnings({"SpellCheckingInspection", "JavadocReference"}) +public class KernelCpuSpeedReader { + private static final String TAG = "KernelCpuSpeedReader"; + + private final String mProcFile; + private final int mNumSpeedSteps; + + public KernelCpuSpeedReader(int cpuNumber, int numSpeedSteps) { + mProcFile = "/sys/devices/system/cpu/cpu" + cpuNumber + "/cpufreq/stats/time_in_state"; + mNumSpeedSteps = numSpeedSteps; + } + + public void smoke() throws IOException { + long[] stepJiffies = readAbsolute(); + if (stepJiffies.length != mNumSpeedSteps) { + throw new IOException("CpuCore Step unmatched, expect = " + mNumSpeedSteps + ", actual = " + stepJiffies.length + ", path = " + mProcFile); + } + } + + public long readTotoal() throws IOException { + long sum = 0; + for (long item : readAbsolute()) { + sum += item; + } + return sum; + } + + /** + * @return The time (in jiffies) spent at different cpu speeds. The values should be + * monotonically increasing, unless the cpu was hotplugged. + */ + public long[] readAbsolute() throws IOException { + long[] speedTimeJiffies = new long[mNumSpeedSteps]; + try (BufferedReader reader = new BufferedReader(new FileReader(mProcFile))) { + TextUtils.SimpleStringSplitter splitter = new TextUtils.SimpleStringSplitter(' '); + String line; + int speedIndex = 0; + while (speedIndex < mNumSpeedSteps && (line = reader.readLine()) != null) { + splitter.setString(line); + splitter.next(); + speedTimeJiffies[speedIndex] = Long.parseLong(splitter.next()); + speedIndex++; + } + } catch (Throwable e) { + throw new IOException("Failed to read cpu-freq: " + e.getMessage(), e); + } + return speedTimeJiffies; + } +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/KernelCpuUidFreqTimeReader.java b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/KernelCpuUidFreqTimeReader.java new file mode 100644 index 000000000..248afed47 --- /dev/null +++ b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/KernelCpuUidFreqTimeReader.java @@ -0,0 +1,102 @@ +package com.tencent.matrix.batterycanary.utils; + + +import android.text.TextUtils; + +import java.io.BufferedReader; +import java.io.FileReader; +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; + +import androidx.annotation.RestrictTo; + +/** + * Reads /proc/uid_time_in_state which has the format: + *

+ * uid: [freq1] [freq2] [freq3] ... + * [uid1]: [time in freq1] [time in freq2] [time in freq3] ... + * [uid2]: [time in freq1] [time in freq2] [time in freq3] ... + * ... + *

+ * This provides the times a UID's processes spent executing at each different cpu frequency. + * The file contains a monotonically increasing count of time for a single boot. This class + * maintains the previous results of a call to {@link #readDelta} in order to provide a proper + * delta. + *

+ * where time is measured in jiffies. + * + * @see com.android.internal.os.KernelCpuUidTimeReader.KernelCpuUidFreqTimeReader + */ +@RestrictTo(RestrictTo.Scope.LIBRARY) +@SuppressWarnings({"SpellCheckingInspection", "JavadocReference"}) +public class KernelCpuUidFreqTimeReader { + private static final String TAG = "KernelCpuUidFreqTimeReader"; + + private final String mProcFile; + private final int[] mClusterSteps; + + public KernelCpuUidFreqTimeReader(int pid, int[] clusterSteps) { + mProcFile = "/proc/" + pid + "/time_in_state"; + mClusterSteps = clusterSteps; + } + + public void smoke() throws IOException { + List cpuCoreStepJiffies = readAbsolute(); + if (mClusterSteps.length != cpuCoreStepJiffies.size()) { + throw new IOException("Cpu clusterNum unmatched, expect = " + mClusterSteps.length + ", actual = " + cpuCoreStepJiffies.size()); + } + for (int i = 0; i < cpuCoreStepJiffies.size(); i++) { + long[] clusterStepJiffies = cpuCoreStepJiffies.get(i); + if (mClusterSteps[i] != clusterStepJiffies.length) { + throw new IOException("Cpu clusterStepNum unmatched, expect = " + mClusterSteps[i] + ", actual = " + clusterStepJiffies.length + ", cluster = " + i); + } + } + } + + public List readTotoal() throws IOException { + List cpuCoreStepJiffies = readAbsolute(); + List cpuCoreJiffies = new ArrayList<>(cpuCoreStepJiffies.size()); + for (long[] stepJiffies : cpuCoreStepJiffies) { + long sum = 0; + for (long item : stepJiffies) { + sum += item; + } + cpuCoreJiffies.add(sum); + } + + return cpuCoreJiffies; + } + + public List readAbsolute() throws IOException { + List cpuCoreJiffies = new ArrayList<>(); + long[] speedJiffies = null; + try (BufferedReader reader = new BufferedReader(new FileReader(mProcFile))) { + TextUtils.SimpleStringSplitter splitter = new TextUtils.SimpleStringSplitter(' '); + String line; + int cluster = -1; + int speedIndex = 0; + while ((line = reader.readLine()) != null) { + if (line.startsWith("cpu")) { + if (cluster >= 0) { + cpuCoreJiffies.add(speedJiffies); + } + cluster++; + speedIndex = 0; + speedJiffies = new long[mClusterSteps[cluster]]; + continue; + } + if (speedIndex < mClusterSteps[cluster]) { + splitter.setString(line); + splitter.next(); + speedJiffies[speedIndex] = Long.parseLong(splitter.next()); + speedIndex++; + } + } + cpuCoreJiffies.add(speedJiffies); + } catch (Throwable e) { + throw new IOException("Failed to read cpu-freq: " + e.getMessage(), e); + } + return cpuCoreJiffies; + } +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/PowerProfile.java b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/PowerProfile.java new file mode 100644 index 000000000..09f6881b2 --- /dev/null +++ b/matrix/matrix-android/matrix-battery-canary/src/main/java/com/tencent/matrix/batterycanary/utils/PowerProfile.java @@ -0,0 +1,504 @@ +package com.tencent.matrix.batterycanary.utils; + +import android.content.Context; +import android.content.res.Resources; +import android.content.res.XmlResourceParser; + +import org.xmlpull.v1.XmlPullParser; +import org.xmlpull.v1.XmlPullParserException; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashMap; + +import androidx.annotation.Nullable; +import androidx.annotation.RestrictTo; + +/** + * @see com.android.internal.os.PowerProfile + */ +@RestrictTo(RestrictTo.Scope.LIBRARY) +@SuppressWarnings({"JavadocReference", "ConstantConditions", "TryFinallyCanBeTryWithResources"}) +public class PowerProfile { + private static PowerProfile sInstance = null; + + @Nullable + public static PowerProfile getInstance() { + return sInstance; + } + + public static PowerProfile init(Context context) throws IOException { + synchronized (sLock) { + try { + sInstance = new PowerProfile(context).smoke(); + return sInstance; + } catch (Throwable e) { + throw new IOException(e); + } + } + } + + public PowerProfile smoke() throws IOException { + if (getNumCpuClusters() <= 0) { + throw new IOException("Invalid cpu clusters: " + getNumCpuClusters()); + } + for (int i = 0; i < getNumCpuClusters(); i++) { + if (getNumSpeedStepsInCpuCluster(i) <= 0) { + throw new IOException("Invalid cpu cluster speed-steps: cluster = " + i + + ", steps = " + getNumSpeedStepsInCpuCluster(i)); + } + } + int cpuCoreNum = BatteryCanaryUtil.getCpuCoreNum(); + int cpuCoreNumInProfile = getCpuCoreNum(); + if (cpuCoreNum != cpuCoreNumInProfile) { + throw new IOException("Unmatched cpu core num, sys = " + cpuCoreNum + + ", profile = " + cpuCoreNumInProfile); + } + return this; + } + + public boolean isSupported() { + try { + smoke(); + return true; + } catch (IOException ignored) { + return false; + } + } + + public int getCpuCoreNum() { + int cpuCoreNumInProfile = 0; + for (int i = 0; i < getNumCpuClusters(); i++) { + cpuCoreNumInProfile += getNumCoresInCpuCluster(i); + } + return cpuCoreNumInProfile; + } + + public int getClusterByCpuNum(int cpuCoreNum) { + int idx = -1; + if (cpuCoreNum < 0) { + return idx; // index out of bound + } + int delta = 0; + for (int i = 0; i < mCpuClusters.length; i++) { + CpuClusterKey cpuCluster = mCpuClusters[i]; + if (cpuCluster.numCpus + delta >= cpuCoreNum + 1) { + return i; + } + delta += cpuCluster.numCpus; + } + return -2; + } + + public static final String POWER_CPU_SUSPEND = "cpu.suspend"; + public static final String POWER_CPU_IDLE = "cpu.idle"; + public static final String POWER_CPU_ACTIVE = "cpu.active"; + + /** + * Power consumption when WiFi driver is scanning for networks. + */ + public static final String POWER_WIFI_SCAN = "wifi.scan"; + + /** + * Power consumption when WiFi driver is on. + */ + public static final String POWER_WIFI_ON = "wifi.on"; + + /** + * Power consumption when WiFi driver is transmitting/receiving. + */ + public static final String POWER_WIFI_ACTIVE = "wifi.active"; + + // + // Updated power constants. These are not estimated, they are real world + // currents and voltages for the underlying bluetooth and wifi controllers. + // + public static final String POWER_WIFI_CONTROLLER_IDLE = "wifi.controller.idle"; + public static final String POWER_WIFI_CONTROLLER_RX = "wifi.controller.rx"; + public static final String POWER_WIFI_CONTROLLER_TX = "wifi.controller.tx"; + public static final String POWER_WIFI_CONTROLLER_TX_LEVELS = "wifi.controller.tx_levels"; + public static final String POWER_WIFI_CONTROLLER_OPERATING_VOLTAGE = "wifi.controller.voltage"; + + public static final String POWER_BLUETOOTH_CONTROLLER_IDLE = "bluetooth.controller.idle"; + public static final String POWER_BLUETOOTH_CONTROLLER_RX = "bluetooth.controller.rx"; + public static final String POWER_BLUETOOTH_CONTROLLER_TX = "bluetooth.controller.tx"; + public static final String POWER_BLUETOOTH_CONTROLLER_OPERATING_VOLTAGE = + "bluetooth.controller.voltage"; + + public static final String POWER_MODEM_CONTROLLER_SLEEP = "modem.controller.sleep"; + public static final String POWER_MODEM_CONTROLLER_IDLE = "modem.controller.idle"; + public static final String POWER_MODEM_CONTROLLER_RX = "modem.controller.rx"; + public static final String POWER_MODEM_CONTROLLER_TX = "modem.controller.tx"; + public static final String POWER_MODEM_CONTROLLER_OPERATING_VOLTAGE = + "modem.controller.voltage"; + + /** + * Power consumption when GPS is on. + */ + public static final String POWER_GPS_ON = "gps.on"; + + /** + * GPS power parameters based on signal quality + */ + public static final String POWER_GPS_SIGNAL_QUALITY_BASED = "gps.signalqualitybased"; + public static final String POWER_GPS_OPERATING_VOLTAGE = "gps.voltage"; + + /** + * Power consumption when Bluetooth driver is on. + * + * @deprecated + */ + @Deprecated + public static final String POWER_BLUETOOTH_ON = "bluetooth.on"; + + /** + * Power consumption when Bluetooth driver is transmitting/receiving. + * + * @deprecated + */ + @Deprecated + public static final String POWER_BLUETOOTH_ACTIVE = "bluetooth.active"; + + /** + * Power consumption when Bluetooth driver gets an AT command. + * + * @deprecated + */ + @Deprecated + public static final String POWER_BLUETOOTH_AT_CMD = "bluetooth.at"; + + /** + * Power consumption when screen is in doze/ambient/always-on mode, including backlight power. + */ + public static final String POWER_AMBIENT_DISPLAY = "ambient.on"; + + /** + * Power consumption when screen is on, not including the backlight power. + */ + public static final String POWER_SCREEN_ON = "screen.on"; + + /** + * Power consumption when cell radio is on but not on a call. + */ + public static final String POWER_RADIO_ON = "radio.on"; + + /** + * Power consumption when cell radio is hunting for a signal. + */ + public static final String POWER_RADIO_SCANNING = "radio.scanning"; + + /** + * Power consumption when talking on the phone. + */ + public static final String POWER_RADIO_ACTIVE = "radio.active"; + + /** + * Power consumption at full backlight brightness. If the backlight is at + * 50% brightness, then this should be multiplied by 0.5 + */ + public static final String POWER_SCREEN_FULL = "screen.full"; + + /** + * Power consumed by the audio hardware when playing back audio content. This is in addition + * to the CPU power, probably due to a DSP and / or amplifier. + */ + public static final String POWER_AUDIO = "audio"; + + /** + * Power consumed by any media hardware when playing back video content. This is in addition + * to the CPU power, probably due to a DSP. + */ + public static final String POWER_VIDEO = "video"; + + /** + * Average power consumption when camera flashlight is on. + */ + public static final String POWER_FLASHLIGHT = "camera.flashlight"; + + /** + * Power consumption when DDR is being used. + */ + public static final String POWER_MEMORY = "memory.bandwidths"; + + /** + * Average power consumption when the camera is on over all standard use cases. + *

+ * TODO: Add more fine-grained camera power metrics. + */ + public static final String POWER_CAMERA = "camera.avg"; + + /** + * Power consumed by wif batched scaning. Broken down into bins by + * Channels Scanned per Hour. May do 1-720 scans per hour of 1-100 channels + * for a range of 1-72,000. Going logrithmic (1-8, 9-64, 65-512, 513-4096, 4097-)! + */ + public static final String POWER_WIFI_BATCHED_SCAN = "wifi.batchedscan"; + + /** + * Battery capacity in milliAmpHour (mAh). + */ + public static final String POWER_BATTERY_CAPACITY = "battery.capacity"; + + /** + * A map from Power Use Item to its power consumption. + */ + static final HashMap sPowerItemMap = new HashMap<>(); + /** + * A map from Power Use Item to an array of its power consumption + * (for items with variable power e.g. CPU). + */ + static final HashMap sPowerArrayMap = new HashMap<>(); + + private static final String TAG_DEVICE = "device"; + private static final String TAG_ITEM = "item"; + private static final String TAG_ARRAY = "array"; + private static final String TAG_ARRAYITEM = "value"; + private static final String ATTR_NAME = "name"; + + private static final Object sLock = new Object(); + + + PowerProfile(Context context) { + // Read the XML file for the given profile (normally only one per device) + synchronized (sLock) { + if (sPowerItemMap.size() == 0 && sPowerArrayMap.size() == 0) { + readPowerValuesFromXml(context); + } + initCpuClusters(); + } + } + + @SuppressWarnings({"ToArrayCallWithZeroLengthArrayArgument", "UnnecessaryBoxing", "CatchMayIgnoreException", "TryWithIdenticalCatches"}) + private void readPowerValuesFromXml(Context context) { + final int id = context.getResources().getIdentifier("power_profile", "xml", "android"); + final Resources resources = context.getResources(); + XmlResourceParser parser = resources.getXml(id); + boolean parsingArray = false; + ArrayList array = new ArrayList<>(); + String arrayName = null; + + try { + XmlUtils.beginDocument(parser, TAG_DEVICE); + + while (true) { + XmlUtils.nextElement(parser); + + String element = parser.getName(); + if (element == null) break; + + if (parsingArray && !element.equals(TAG_ARRAYITEM)) { + // Finish array + sPowerArrayMap.put(arrayName, array.toArray(new Double[array.size()])); + parsingArray = false; + } + if (element.equals(TAG_ARRAY)) { + parsingArray = true; + array.clear(); + arrayName = parser.getAttributeValue(null, ATTR_NAME); + } else if (element.equals(TAG_ITEM) || element.equals(TAG_ARRAYITEM)) { + String name = null; + if (!parsingArray) name = parser.getAttributeValue(null, ATTR_NAME); + if (parser.next() == XmlPullParser.TEXT) { + String power = parser.getText(); + double value = 0; + try { + value = Double.valueOf(power); + } catch (NumberFormatException nfe) { + } + if (element.equals(TAG_ITEM)) { + sPowerItemMap.put(name, value); + } else if (parsingArray) { + array.add(value); + } + } + } + } + if (parsingArray) { + sPowerArrayMap.put(arrayName, array.toArray(new Double[array.size()])); + } + } catch (XmlPullParserException e) { + throw new RuntimeException(e); + } catch (IOException e) { + throw new RuntimeException(e); + } finally { + parser.close(); + } + + // Now collect other config variables. + int[] configResIds = new int[]{ + context.getResources().getIdentifier("config_bluetooth_idle_cur_ma", "integer", "android"), + context.getResources().getIdentifier("config_bluetooth_rx_cur_ma", "integer", "android"), + context.getResources().getIdentifier("config_bluetooth_tx_cur_ma", "integer", "android"), + context.getResources().getIdentifier("config_bluetooth_operating_voltage_mv", "integer", "android"), + }; + + String[] configResIdKeys = new String[]{ + POWER_BLUETOOTH_CONTROLLER_IDLE, + POWER_BLUETOOTH_CONTROLLER_RX, + POWER_BLUETOOTH_CONTROLLER_TX, + POWER_BLUETOOTH_CONTROLLER_OPERATING_VOLTAGE, + }; + + for (int i = 0; i < configResIds.length; i++) { + String key = configResIdKeys[i]; + // if we already have some of these parameters in power_profile.xml, ignore the + // value in config.xml + if ((sPowerItemMap.containsKey(key) && sPowerItemMap.get(key) > 0)) { + continue; + } + int value = resources.getInteger(configResIds[i]); + if (value > 0) { + sPowerItemMap.put(key, (double) value); + } + } + } + + private CpuClusterKey[] mCpuClusters; + + private static final String CPU_PER_CLUSTER_CORE_COUNT = "cpu.clusters.cores"; + private static final String CPU_CLUSTER_POWER_COUNT = "cpu.cluster_power.cluster"; + private static final String CPU_CORE_SPEED_PREFIX = "cpu.core_speeds.cluster"; + private static final String CPU_CORE_POWER_PREFIX = "cpu.core_power.cluster"; + + private void initCpuClusters() { + if (sPowerArrayMap.containsKey(CPU_PER_CLUSTER_CORE_COUNT)) { + final Double[] data = sPowerArrayMap.get(CPU_PER_CLUSTER_CORE_COUNT); + mCpuClusters = new CpuClusterKey[data.length]; + for (int cluster = 0; cluster < data.length; cluster++) { + int numCpusInCluster = (int) Math.round(data[cluster]); + mCpuClusters[cluster] = new CpuClusterKey( + CPU_CORE_SPEED_PREFIX + cluster, CPU_CLUSTER_POWER_COUNT + cluster, + CPU_CORE_POWER_PREFIX + cluster, numCpusInCluster); + } + } else { + // Default to single. + mCpuClusters = new CpuClusterKey[1]; + int numCpus = 1; + if (sPowerItemMap.containsKey(CPU_PER_CLUSTER_CORE_COUNT)) { + numCpus = (int) Math.round(sPowerItemMap.get(CPU_PER_CLUSTER_CORE_COUNT)); + } + mCpuClusters[0] = new CpuClusterKey(CPU_CORE_SPEED_PREFIX + 0, + CPU_CLUSTER_POWER_COUNT + 0, CPU_CORE_POWER_PREFIX + 0, numCpus); + } + } + + public static class CpuClusterKey { + private final String freqKey; + private final String clusterPowerKey; + private final String corePowerKey; + private final int numCpus; + + private CpuClusterKey(String freqKey, String clusterPowerKey, + String corePowerKey, int numCpus) { + this.freqKey = freqKey; + this.clusterPowerKey = clusterPowerKey; + this.corePowerKey = corePowerKey; + this.numCpus = numCpus; + } + } + + public int getNumCpuClusters() { + return mCpuClusters.length; + } + + public int getNumCoresInCpuCluster(int cluster) { + return mCpuClusters[cluster].numCpus; + } + + public int getNumSpeedStepsInCpuCluster(int cluster) { + if (cluster < 0 || cluster >= mCpuClusters.length) { + return 0; // index out of bound + } + if (sPowerArrayMap.containsKey(mCpuClusters[cluster].freqKey)) { + return sPowerArrayMap.get(mCpuClusters[cluster].freqKey).length; + } + return 1; // Only one speed + } + + public double getAveragePowerForCpuCluster(int cluster) { + if (cluster >= 0 && cluster < mCpuClusters.length) { + return getAveragePower(mCpuClusters[cluster].clusterPowerKey); + } + return 0; + } + + public double getAveragePowerForCpuCore(int cluster, int step) { + if (cluster >= 0 && cluster < mCpuClusters.length) { + return getAveragePower(mCpuClusters[cluster].corePowerKey, step); + } + return 0; + } + + public int getNumElements(String key) { + if (sPowerItemMap.containsKey(key)) { + return 1; + } else if (sPowerArrayMap.containsKey(key)) { + return sPowerArrayMap.get(key).length; + } + return 0; + } + + + public double getAveragePowerOrDefault(String type, double defaultValue) { + if (sPowerItemMap.containsKey(type)) { + return sPowerItemMap.get(type); + } else if (sPowerArrayMap.containsKey(type)) { + return sPowerArrayMap.get(type)[0]; + } else { + return defaultValue; + } + } + + + public double getAveragePower(String type) { + return getAveragePowerOrDefault(type, 0); + } + + + public double getAveragePower(String type, int level) { + if (sPowerItemMap.containsKey(type)) { + return sPowerItemMap.get(type); + } else if (sPowerArrayMap.containsKey(type)) { + final Double[] values = sPowerArrayMap.get(type); + if (values.length > level && level >= 0) { + return values[level]; + } else if (level < 0 || values.length == 0) { + return 0; + } else { + return values[values.length - 1]; + } + } else { + return 0; + } + } + + public double getBatteryCapacity() { + return getAveragePower(POWER_BATTERY_CAPACITY); + } + + @SuppressWarnings("StatementWithEmptyBody") + static class XmlUtils { + public static void beginDocument(XmlPullParser parser, String firstElementName) throws XmlPullParserException, IOException { + int type; + while ((type = parser.next()) != parser.START_TAG + && type != parser.END_DOCUMENT) { + } + + if (type != parser.START_TAG) { + throw new XmlPullParserException("No start tag found"); + } + + if (!parser.getName().equals(firstElementName)) { + throw new XmlPullParserException("Unexpected start tag: found " + parser.getName() + + ", expected " + firstElementName); + } + } + + public static void nextElement(XmlPullParser parser) throws XmlPullParserException, IOException { + int type; + while ((type = parser.next()) != parser.START_TAG + && type != parser.END_DOCUMENT) { + } + } + } +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-commons/src/main/java/com/android/dexdeps/DexData.java b/matrix/matrix-android/matrix-commons/src/main/java/com/android/dexdeps/DexData.java index 9ca5f9daf..d3dbbd7fa 100644 --- a/matrix/matrix-android/matrix-commons/src/main/java/com/android/dexdeps/DexData.java +++ b/matrix/matrix-android/matrix-commons/src/main/java/com/android/dexdeps/DexData.java @@ -70,7 +70,9 @@ public void load() throws IOException { private static boolean verifyMagic(byte[] magic) { return Arrays.equals(magic, HeaderItem.DEX_FILE_MAGIC_v035) || Arrays.equals(magic, HeaderItem.DEX_FILE_MAGIC_v037) - || Arrays.equals(magic, HeaderItem.DEX_FILE_MAGIC_v038); + || Arrays.equals(magic, HeaderItem.DEX_FILE_MAGIC_v038) + || Arrays.equals(magic, HeaderItem.DEX_FILE_MAGIC_v039) + || Arrays.equals(magic, HeaderItem.DEX_FILE_MAGIC_v040); } /** @@ -663,6 +665,14 @@ static class HeaderItem { public static final byte[] DEX_FILE_MAGIC_v038 = "dex\n038\0".getBytes(StandardCharsets.US_ASCII); + // Dex version 039: Android "P" and beyond. + public static final byte[] DEX_FILE_MAGIC_v039 = + "dex\n039\0".getBytes(StandardCharsets.US_ASCII); + + // Dex version 040: beyond Android "10" (previously known as Android "Q"). + public static final byte[] DEX_FILE_MAGIC_v040 = + "dex\n040\0".getBytes(StandardCharsets.US_ASCII); + public static final int ENDIAN_CONSTANT = 0x12345678; public static final int REVERSE_ENDIAN_CONSTANT = 0x78563412; } diff --git a/matrix/matrix-android/matrix-commons/src/main/java/com/tencent/matrix/javalib/util/FileUtil.java b/matrix/matrix-android/matrix-commons/src/main/java/com/tencent/matrix/javalib/util/FileUtil.java index 648a71cef..0358b8908 100644 --- a/matrix/matrix-android/matrix-commons/src/main/java/com/tencent/matrix/javalib/util/FileUtil.java +++ b/matrix/matrix-android/matrix-commons/src/main/java/com/tencent/matrix/javalib/util/FileUtil.java @@ -252,6 +252,13 @@ public static void unzip(String filePath, String destFolder) { Enumeration emu = zipFile.entries(); while (emu.hasMoreElements()) { ZipEntry entry = (ZipEntry) emu.nextElement(); + String entryName = entry.getName(); + + if (Util.preventZipSlip(new File(destFolder), entryName)) { + Log.e(TAG, "writeEntry entry %s failed!", entryName); + continue; + } + if (entry.isDirectory()) { new File(destFolder, entry.getName()).mkdirs(); continue; diff --git a/matrix/matrix-android/matrix-commons/src/main/java/com/tencent/matrix/javalib/util/Util.java b/matrix/matrix-android/matrix-commons/src/main/java/com/tencent/matrix/javalib/util/Util.java index e27dc7e0d..98fbddca9 100644 --- a/matrix/matrix-android/matrix-commons/src/main/java/com/tencent/matrix/javalib/util/Util.java +++ b/matrix/matrix-android/matrix-commons/src/main/java/com/tencent/matrix/javalib/util/Util.java @@ -16,6 +16,8 @@ package com.tencent.matrix.javalib.util; +import java.io.File; +import java.io.IOException; import java.util.regex.Pattern; /** @@ -35,6 +37,19 @@ public static String nullAsNil(String str) { return str == null ? "" : str; } + public static boolean preventZipSlip(java.io.File output, String zipEntryName) { + + try { + if (zipEntryName.contains("..") && new File(output, zipEntryName).getCanonicalPath().startsWith(output.getCanonicalPath() + File.separator)) { + return true; + } + } catch (IOException e) { + e.printStackTrace(); + return true; + } + return false; + } + public static boolean isNumber(String str) { Pattern pattern = Pattern.compile("\\d+"); return pattern.matcher(str).matches(); diff --git a/matrix/matrix-android/matrix-gradle-plugin/build.gradle b/matrix/matrix-android/matrix-gradle-plugin/build.gradle index 7807c7d83..76a401aa8 100644 --- a/matrix/matrix-android/matrix-gradle-plugin/build.gradle +++ b/matrix/matrix-android/matrix-gradle-plugin/build.gradle @@ -5,8 +5,8 @@ dependencies { implementation fileTree(include: ['*.jar'], dir: 'libs') implementation gradleApi() implementation project(':matrix-commons') - implementation group: 'org.ow2.asm', name: 'asm', version: '5.1' - implementation group: 'org.ow2.asm', name: 'asm-commons', version: '5.1' + implementation group: 'org.ow2.asm', name: 'asm', version: '7.0' + implementation group: 'org.ow2.asm', name: 'asm-commons', version: '7.0' implementation 'com.android.tools.build:gradle:4.0.0' implementation project(':matrix-arscutil') implementation "org.jetbrains.kotlin:kotlin-stdlib:${gradle.KOTLIN_VERSION}" diff --git a/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/Configuration.java b/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/Configuration.java index c4819f6c8..8fb16b8d4 100644 --- a/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/Configuration.java +++ b/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/Configuration.java @@ -15,13 +15,14 @@ public class Configuration { public String ignoreMethodMapFilePath; public String blockListFilePath; public String traceClassOut; + public boolean skipCheckClass; public HashSet blockSet = new HashSet<>(); public Configuration() { } Configuration(String packageName, String mappingDir, String baseMethodMapPath, String methodMapFilePath, - String ignoreMethodMapFilePath, String blockListFilePath, String traceClassOut) { + String ignoreMethodMapFilePath, String blockListFilePath, String traceClassOut, boolean skipCheckClass) { this.packageName = packageName; this.mappingDir = Util.nullAsNil(mappingDir); this.baseMethodMapPath = Util.nullAsNil(baseMethodMapPath); @@ -29,6 +30,7 @@ public Configuration() { this.ignoreMethodMapFilePath = Util.nullAsNil(ignoreMethodMapFilePath); this.blockListFilePath = Util.nullAsNil(blockListFilePath); this.traceClassOut = Util.nullAsNil(traceClassOut); + this.skipCheckClass = skipCheckClass; } public int parseBlockFile(MappingCollector processor) { @@ -82,6 +84,7 @@ public static class Builder { public String ignoreMethodMapFile; public String blockListFile; public String traceClassOut; + public boolean skipCheckClass = false; public Builder setPackageName(String packageName) { this.packageName = packageName; @@ -118,8 +121,13 @@ public Builder setBlockListFile(String blockListFile) { return this; } + public Builder setSkipCheckClass(boolean skipCheckClass) { + this.skipCheckClass = skipCheckClass; + return this; + } + public Configuration build() { - return new Configuration(packageName, mappingPath, baseMethodMap, methodMapFile, ignoreMethodMapFile, blockListFile, traceClassOut); + return new Configuration(packageName, mappingPath, baseMethodMap, methodMapFile, ignoreMethodMapFile, blockListFile, traceClassOut, skipCheckClass); } } diff --git a/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/MethodCollector.java b/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/MethodCollector.java index 00d6c9ccd..66bca5f42 100644 --- a/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/MethodCollector.java +++ b/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/MethodCollector.java @@ -1,6 +1,7 @@ package com.tencent.matrix.trace; import com.tencent.matrix.javalib.util.Log; +import com.tencent.matrix.plugin.compat.AgpCompat; import com.tencent.matrix.trace.item.TraceMethod; import com.tencent.matrix.trace.retrace.MappingCollector; @@ -131,7 +132,7 @@ public void run() { is = new FileInputStream(classFile); ClassReader classReader = new ClassReader(is); ClassWriter classWriter = new ClassWriter(ClassWriter.COMPUTE_MAXS); - ClassVisitor visitor = new TraceClassAdapter(Opcodes.ASM5, classWriter); + ClassVisitor visitor = new TraceClassAdapter(AgpCompat.getAsmApi(), classWriter); classReader.accept(visitor, 0); } catch (Exception e) { @@ -167,7 +168,7 @@ public void run() { InputStream inputStream = zipFile.getInputStream(zipEntry); ClassReader classReader = new ClassReader(inputStream); ClassWriter classWriter = new ClassWriter(ClassWriter.COMPUTE_MAXS); - ClassVisitor visitor = new TraceClassAdapter(Opcodes.ASM5, classWriter); + ClassVisitor visitor = new TraceClassAdapter(AgpCompat.getAsmApi(), classWriter); classReader.accept(visitor, 0); } } @@ -305,7 +306,7 @@ private class CollectMethodNode extends MethodNode { CollectMethodNode(String className, int access, String name, String desc, String signature, String[] exceptions) { - super(Opcodes.ASM5, access, name, desc, signature, exceptions); + super(AgpCompat.getAsmApi(), access, name, desc, signature, exceptions); this.className = className; } diff --git a/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/MethodTracer.java b/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/MethodTracer.java index 1084560dd..8c43c72bd 100644 --- a/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/MethodTracer.java +++ b/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/MethodTracer.java @@ -18,6 +18,8 @@ import com.tencent.matrix.javalib.util.FileUtil; import com.tencent.matrix.javalib.util.Log; +import com.tencent.matrix.javalib.util.Util; +import com.tencent.matrix.plugin.compat.AgpCompat; import com.tencent.matrix.trace.item.TraceMethod; import com.tencent.matrix.trace.retrace.MappingCollector; @@ -27,6 +29,7 @@ import org.objectweb.asm.MethodVisitor; import org.objectweb.asm.Opcodes; import org.objectweb.asm.commons.AdviceAdapter; +import org.objectweb.asm.util.CheckClassAdapter; import java.io.ByteArrayInputStream; import java.io.File; @@ -67,6 +70,8 @@ public class MethodTracer { private final ExecutorService executor; private MappingCollector mappingCollector; + private volatile boolean traceError = false; + public MethodTracer(ExecutorService executor, MappingCollector mappingCollector, Configuration config, ConcurrentHashMap collectedMap, ConcurrentHashMap collectedClassExtendMap) { this.configuration = config; this.mappingCollector = mappingCollector; @@ -75,44 +80,46 @@ public MethodTracer(ExecutorService executor, MappingCollector mappingCollector, this.collectedMethodMap = collectedMap; } - - public void trace(Map srcFolderList, Map dependencyJarList) throws ExecutionException, InterruptedException { + public void trace(Map srcFolderList, Map dependencyJarList, ClassLoader classLoader, boolean ignoreCheckClass) throws ExecutionException, InterruptedException { List futures = new LinkedList<>(); - traceMethodFromSrc(srcFolderList, futures); - traceMethodFromJar(dependencyJarList, futures); + traceMethodFromSrc(srcFolderList, futures, classLoader, ignoreCheckClass); + traceMethodFromJar(dependencyJarList, futures, classLoader, ignoreCheckClass); for (Future future : futures) { future.get(); } + if (traceError) { + throw new IllegalArgumentException("something wrong with trace, see detail log before"); + } futures.clear(); } - private void traceMethodFromSrc(Map srcMap, List futures) { + private void traceMethodFromSrc(Map srcMap, List futures, final ClassLoader classLoader, final boolean skipCheckClass) { if (null != srcMap) { for (Map.Entry entry : srcMap.entrySet()) { futures.add(executor.submit(new Runnable() { @Override public void run() { - innerTraceMethodFromSrc(entry.getKey(), entry.getValue()); + innerTraceMethodFromSrc(entry.getKey(), entry.getValue(), classLoader, skipCheckClass); } })); } } } - private void traceMethodFromJar(Map dependencyMap, List futures) { + private void traceMethodFromJar(Map dependencyMap, List futures, final ClassLoader classLoader, final boolean skipCheckClass) { if (null != dependencyMap) { for (Map.Entry entry : dependencyMap.entrySet()) { futures.add(executor.submit(new Runnable() { @Override public void run() { - innerTraceMethodFromJar(entry.getKey(), entry.getValue()); + innerTraceMethodFromJar(entry.getKey(), entry.getValue(), classLoader, skipCheckClass); } })); } } } - private void innerTraceMethodFromSrc(File input, File output) { + private void innerTraceMethodFromSrc(File input, File output, ClassLoader classLoader, boolean ignoreCheckClass) { ArrayList classFileList = new ArrayList<>(); if (input.isDirectory()) { @@ -133,25 +140,40 @@ private void innerTraceMethodFromSrc(File input, File output) { changedFileOutput.createNewFile(); if (MethodCollector.isNeedTraceFile(classFile.getName())) { + is = new FileInputStream(classFile); ClassReader classReader = new ClassReader(is); - ClassWriter classWriter = new ClassWriter(ClassWriter.COMPUTE_MAXS); - ClassVisitor classVisitor = new TraceClassAdapter(Opcodes.ASM5, classWriter); + ClassWriter classWriter = new TraceClassWriter(ClassWriter.COMPUTE_FRAMES, classLoader); + ClassVisitor classVisitor = new TraceClassAdapter(AgpCompat.getAsmApi(), classWriter); classReader.accept(classVisitor, ClassReader.EXPAND_FRAMES); is.close(); + byte[] data = classWriter.toByteArray(); + + if (!ignoreCheckClass) { + try { + ClassReader cr = new ClassReader(data); + ClassWriter cw = new ClassWriter(0); + ClassVisitor check = new CheckClassAdapter(cw); + cr.accept(check, ClassReader.EXPAND_FRAMES); + } catch (Throwable e) { + System.err.println("trace output ERROR : " + e.getMessage() + ", " + classFile); + traceError = true; + } + } + if (output.isDirectory()) { os = new FileOutputStream(changedFileOutput); } else { os = new FileOutputStream(output); } - os.write(classWriter.toByteArray()); + os.write(data); os.close(); } else { FileUtil.copyFileUsingStream(classFile, changedFileOutput); } } catch (Exception e) { - Log.e(TAG, "[innerTraceMethodFromSrc] input:%s e:%s", input.getName(), e); + Log.e(TAG, "[innerTraceMethodFromSrc] input:%s e:%s", input.getName(), e.getMessage()); try { Files.copy(input.toPath(), output.toPath(), StandardCopyOption.REPLACE_EXISTING); } catch (Exception e1) { @@ -168,7 +190,7 @@ private void innerTraceMethodFromSrc(File input, File output) { } } - private void innerTraceMethodFromJar(File input, File output) { + private void innerTraceMethodFromJar(File input, File output, final ClassLoader classLoader, boolean skipCheckClass) { ZipOutputStream zipOutputStream = null; ZipFile zipFile = null; try { @@ -178,13 +200,34 @@ private void innerTraceMethodFromJar(File input, File output) { while (enumeration.hasMoreElements()) { ZipEntry zipEntry = enumeration.nextElement(); String zipEntryName = zipEntry.getName(); + + if (Util.preventZipSlip(output, zipEntryName)) { + Log.e(TAG, "Unzip entry %s failed!", zipEntryName); + continue; + } + if (MethodCollector.isNeedTraceFile(zipEntryName)) { + InputStream inputStream = zipFile.getInputStream(zipEntry); ClassReader classReader = new ClassReader(inputStream); - ClassWriter classWriter = new ClassWriter(ClassWriter.COMPUTE_MAXS); - ClassVisitor classVisitor = new TraceClassAdapter(Opcodes.ASM5, classWriter); + ClassWriter classWriter = new TraceClassWriter(ClassWriter.COMPUTE_FRAMES, classLoader); + ClassVisitor classVisitor = new TraceClassAdapter(AgpCompat.getAsmApi(), classWriter); classReader.accept(classVisitor, ClassReader.EXPAND_FRAMES); byte[] data = classWriter.toByteArray(); +// + if (!skipCheckClass) { + try { + ClassReader r = new ClassReader(data); + ClassWriter w = new ClassWriter(0); + ClassVisitor v = new CheckClassAdapter(w); + r.accept(v, ClassReader.EXPAND_FRAMES); + } catch (Throwable e) { + System.err.println("trace jar output ERROR: " + e.getMessage() + ", " + zipEntryName); +// e.printStackTrace(); + traceError = true; + } + } + InputStream byteArrayInputStream = new ByteArrayInputStream(data); ZipEntry newZipEntry = new ZipEntry(zipEntryName); FileUtil.addZipEntry(zipOutputStream, newZipEntry, byteArrayInputStream); @@ -195,7 +238,7 @@ private void innerTraceMethodFromJar(File input, File output) { } } } catch (Exception e) { - Log.e(TAG, "[innerTraceMethodFromJar] input:%s output:%s e:%s", input, output, e); + Log.e(TAG, "[innerTraceMethodFromJar] input:%s output:%s e:%s", input, output, e.getMessage()); if (e instanceof ZipException) { e.printStackTrace(); } diff --git a/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/TraceClassLoader.java b/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/TraceClassLoader.java new file mode 100644 index 000000000..64bde9ed6 --- /dev/null +++ b/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/TraceClassLoader.java @@ -0,0 +1,70 @@ +package com.tencent.matrix.trace; + +import com.android.build.gradle.AppExtension; +import com.android.build.gradle.BaseExtension; +import com.android.build.gradle.LibraryExtension; +import com.google.common.collect.ImmutableList; + +import org.gradle.api.Project; + +import java.io.File; +import java.net.MalformedURLException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.Collection; + +/** + * Created by habbyge on 2019/4/24. + */ +public class TraceClassLoader { + + public static URLClassLoader getClassLoader(Project project, Collection inputFiles) + throws MalformedURLException { + + ImmutableList.Builder urls = new ImmutableList.Builder<>(); + File androidJar = getAndroidJar(project); + if (androidJar != null) { + urls.add(androidJar.toURI().toURL()); + } + + for (File inputFile : inputFiles) { + urls.add(inputFile.toURI().toURL()); + } + +// for (TransformInput inputs : Iterables.concat(invocation.getInputs(), invocation.getReferencedInputs())) { +// for (DirectoryInput directoryInput : inputs.getDirectoryInputs()) { +// if (directoryInput.getFile().isDirectory()) { +// urls.add(directoryInput.getFile().toURI().toURL()); +// } +// } +// for (JarInput jarInput : inputs.getJarInputs()) { +// if (jarInput.getFile().isFile()) { +// urls.add(jarInput.getFile().toURI().toURL()); +// } +// } +// } + + ImmutableList urlImmutableList = urls.build(); + URL[] classLoaderUrls = urlImmutableList.toArray(new URL[urlImmutableList.size()]); + return new URLClassLoader(classLoaderUrls); + } + + private static File getAndroidJar(Project project) { + BaseExtension extension = null; + if (project.getPlugins().hasPlugin("com.android.application")) { + extension = project.getExtensions().findByType(AppExtension.class); + } else if (project.getPlugins().hasPlugin("com.android.library")) { + extension = project.getExtensions().findByType(LibraryExtension.class); + } + if (extension == null) { + return null; + } + + String sdkDirectory = extension.getSdkDirectory().getAbsolutePath(); + String compileSdkVersion = extension.getCompileSdkVersion(); + sdkDirectory = sdkDirectory + File.separator + "platforms" + File.separator; + String androidJarPath = sdkDirectory + compileSdkVersion + File.separator + "android.jar"; + File androidJar = new File(androidJarPath); + return androidJar.exists() ? androidJar : null; + } +} diff --git a/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/TraceClassWriter.java b/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/TraceClassWriter.java new file mode 100644 index 000000000..38d12cb1e --- /dev/null +++ b/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/TraceClassWriter.java @@ -0,0 +1,152 @@ +package com.tencent.matrix.trace; + +import org.objectweb.asm.ClassReader; +import org.objectweb.asm.ClassWriter; + +/** + * Created by habbyge on 2019/4/25. + * + * fix: + * java.lang.TypeNotPresentException: Type android/content/res/TypedArray not present + * at org.objectweb.asm.ClassWriter.getCommonSuperClass(ClassWriter.java:1025) + * at org.objectweb.asm.SymbolTable.addMergedType(SymbolTable.java:1202) + * at org.objectweb.asm.Frame.merge(Frame.java:1299) + * at org.objectweb.asm.Frame.merge(Frame.java:1197) + * at org.objectweb.asm.MethodWriter.computeAllFrames(MethodWriter.java:1610) + * at org.objectweb.asm.MethodWriter.visitMaxs(MethodWriter.java:1546) + * at org.objectweb.asm.tree.MethodNode.accept(MethodNode.java:769) + * at org.objectweb.asm.util.CheckMethodAdapter$1.visitEnd(CheckMethodAdapter.java:465) + * at org.objectweb.asm.MethodVisitor.visitEnd(MethodVisitor.java:783) + * at org.objectweb.asm.util.CheckMethodAdapter.visitEnd(CheckMethodAdapter.java:1036) + * at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1495) + * at org.objectweb.asm.ClassReader.accept(ClassReader.java:721) + * + */ +class TraceClassWriter extends ClassWriter { + private ClassLoader mClassLoader; + TraceClassWriter(int flags, ClassLoader classLoader) { + super(flags); + mClassLoader = classLoader; + } + + TraceClassWriter(ClassReader classReader, int flags, ClassLoader classLoader) { + super(classReader, flags); + mClassLoader = classLoader; + } + + @Override + protected String getCommonSuperClass(String type1, String type2) { + try { + return super.getCommonSuperClass(type1, type2); + } catch (Exception e) { + try { + return getCommonSuperClassV1(type1, type2); + } catch (Exception e1) { + try { + return getCommonSuperClassV2(type1, type2); + } catch (Exception e2) { + return getCommonSuperClassV3(type1, type2); + } + } + } + } + + private String getCommonSuperClassV1(String type1, String type2) { + ClassLoader curClassLoader = getClass().getClassLoader(); + + Class clazz1, clazz2; + try { + clazz1 = Class.forName(type1.replace('/', '.'), false, mClassLoader); + clazz2 = Class.forName(type2.replace('/', '.'), false, mClassLoader); + } catch (Exception e) { + /*throw new RuntimeException(e.toString());*/ + /*e.printStackTrace();*/ + return "java/lang/Object"; + } catch (LinkageError error) { + return "java/lang/Object"; + } + //noinspection unchecked + if (clazz1.isAssignableFrom(clazz2)) { + return type1; + } + //noinspection unchecked + if (clazz2.isAssignableFrom(clazz1)) { + return type2; + } + if (clazz1.isInterface() || clazz2.isInterface()) { + return "java/lang/Object"; + } else { + //noinspection unchecked + do { + clazz1 = clazz1.getSuperclass(); + } while (!clazz1.isAssignableFrom(clazz2)); + return clazz1.getName().replace('.', '/'); + } + } + + private String getCommonSuperClassV2(String type1, String type2) { + ClassLoader curClassLoader = getClass().getClassLoader(); + + Class clazz1, clazz2; + try { + clazz1 = Class.forName(type1.replace('/', '.'), false, curClassLoader); + clazz2 = Class.forName(type2.replace('/', '.'), false, mClassLoader); + } catch (Exception e) { + /*throw new RuntimeException(e.toString());*/ + /*e.printStackTrace();*/ + return "java/lang/Object"; + } catch (LinkageError error) { + return "java/lang/Object"; + } + //noinspection unchecked + if (clazz1.isAssignableFrom(clazz2)) { + return type1; + } + //noinspection unchecked + if (clazz2.isAssignableFrom(clazz1)) { + return type2; + } + if (clazz1.isInterface() || clazz2.isInterface()) { + return "java/lang/Object"; + } else { + //noinspection unchecked + do { + clazz1 = clazz1.getSuperclass(); + } while (!clazz1.isAssignableFrom(clazz2)); + return clazz1.getName().replace('.', '/'); + } + } + + private String getCommonSuperClassV3(String type1, String type2) { + ClassLoader curClassLoader = getClass().getClassLoader(); + + Class clazz1, clazz2; + try { + clazz1 = Class.forName(type1.replace('/', '.'), false, mClassLoader); + clazz2 = Class.forName(type2.replace('/', '.'), false, curClassLoader); + } catch (Exception e) { + /*e.printStackTrace();*/ + /*throw new RuntimeException(e.toString());*/ + return "java/lang/Object"; + } catch (LinkageError error) { + return "java/lang/Object"; + } + //noinspection unchecked + if (clazz1.isAssignableFrom(clazz2)) { + return type1; + } + //noinspection unchecked + if (clazz2.isAssignableFrom(clazz1)) { + return type2; + } + if (clazz1.isInterface() || clazz2.isInterface()) { + return "java/lang/Object"; + } else { + //noinspection unchecked + do { + clazz1 = clazz1.getSuperclass(); + } while (!clazz1.isAssignableFrom(clazz2)); + return clazz1.getName().replace('.', '/'); + } + } +} diff --git a/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/extension/MatrixTraceExtension.java b/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/extension/MatrixTraceExtension.java index 95a42b37b..8261d741e 100644 --- a/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/extension/MatrixTraceExtension.java +++ b/matrix/matrix-android/matrix-gradle-plugin/src/main/java/com/tencent/matrix/trace/extension/MatrixTraceExtension.java @@ -5,6 +5,7 @@ public class MatrixTraceExtension { String baseMethodMapFile; String blackListFile; String customDexTransformName; + boolean skipCheckClass = true; // skip by default boolean enable; @@ -32,4 +33,8 @@ public boolean isTransformInjectionForced() { public boolean isEnable() { return enable; } + + public boolean isSkipCheckClass() { + return skipCheckClass; + } } diff --git a/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/compat/AgpCompat.kt b/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/compat/AgpCompat.kt index 71ed7f270..a5a962181 100644 --- a/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/compat/AgpCompat.kt +++ b/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/compat/AgpCompat.kt @@ -17,7 +17,9 @@ package com.tencent.matrix.plugin.compat import com.android.build.gradle.api.BaseVariant +import com.android.build.gradle.internal.api.ReadOnlyBuildType import com.android.builder.model.SigningConfig +import org.objectweb.asm.Opcodes class AgpCompat { @@ -32,8 +34,15 @@ class AgpCompat { } fun getSigningConfig(variant: BaseVariant): SigningConfig? { - return variant.buildType.signingConfig + return (variant.buildType as ReadOnlyBuildType).signingConfig } + + @JvmStatic + val asmApi: Int + get() = when { + VersionsCompat.greatThanOrEqual(AGPVersion.AGP_7_0_0) -> Opcodes.ASM6 + else -> Opcodes.ASM5 + } } } \ No newline at end of file diff --git a/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/compat/VersionsCompat.kt b/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/compat/VersionsCompat.kt index b1c95cb23..9d1ca9530 100644 --- a/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/compat/VersionsCompat.kt +++ b/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/compat/VersionsCompat.kt @@ -24,6 +24,7 @@ enum class AGPVersion( AGP_3_6_0("3.6.0"), AGP_4_0_0("4.0.0"), AGP_4_1_0("4.1.0"), + AGP_7_0_0("7.0.0") } class VersionsCompat { diff --git a/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/task/MatrixTraceTask.kt b/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/task/MatrixTraceTask.kt index ad72706fc..e7c52434f 100644 --- a/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/task/MatrixTraceTask.kt +++ b/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/task/MatrixTraceTask.kt @@ -78,6 +78,9 @@ abstract class MatrixTraceTask : DefaultTask() { @get:Optional abstract val methodMapFileOutput: RegularFileProperty + @get:Optional + abstract val skipCheckClass: Property + @TaskAction fun execute(inputChanges: InputChanges) { @@ -107,11 +110,13 @@ abstract class MatrixTraceTask : DefaultTask() { methodMapFilePath = methodMapFileOutput.asFile.get().absolutePath, baseMethodMapPath = baseMethodMapFile.asFile.orNull?.absolutePath, blockListFilePath = blockListFile.asFile.orNull?.absolutePath, - mappingDir = mappingDir.get() + mappingDir = mappingDir.get(), + project = project ).doTransform( classInputs = classInputs.files, changedFiles = changedFiles, isIncremental = incremental, + skipCheckClass = this.skipCheckClass.get(), traceClassDirectoryOutput = outputDirectory, inputToOutput = ConcurrentHashMap(), legacyReplaceChangedFile = null, @@ -173,6 +178,7 @@ abstract class MatrixTraceTask : DefaultTask() { } task.mappingDir.set(mappingOut) task.traceClassOutputDirectory.set(traceClassOut) + task.skipCheckClass.set(extension.isSkipCheckClass) task.classOutputs.from(project.files(Callable> { val outputDirectory = File(task.traceClassOutputDirectory.get()) diff --git a/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/trace/MatrixTrace.kt b/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/trace/MatrixTrace.kt index 3c6ab5c11..27a3ebc80 100644 --- a/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/trace/MatrixTrace.kt +++ b/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/trace/MatrixTrace.kt @@ -22,13 +22,11 @@ import com.google.common.hash.Hashing import com.tencent.matrix.javalib.util.IOUtil import com.tencent.matrix.javalib.util.Log import com.tencent.matrix.javalib.util.Util -import com.tencent.matrix.trace.Configuration -import com.tencent.matrix.trace.MethodCollector -import com.tencent.matrix.trace.MethodTracer -import com.tencent.matrix.trace.TraceBuildConstants +import com.tencent.matrix.trace.* import com.tencent.matrix.trace.item.TraceMethod import com.tencent.matrix.trace.retrace.MappingCollector import com.tencent.matrix.trace.retrace.MappingReader +import org.gradle.api.Project import java.io.File import java.util.* import java.util.concurrent.ConcurrentHashMap @@ -36,6 +34,7 @@ import java.util.concurrent.ExecutorService import java.util.concurrent.Executors import java.util.concurrent.Future import java.util.concurrent.atomic.AtomicInteger +import kotlin.collections.ArrayList import kotlin.collections.HashMap class MatrixTrace( @@ -43,7 +42,8 @@ class MatrixTrace( private val methodMapFilePath: String, private val baseMethodMapPath: String?, private val blockListFilePath: String?, - private val mappingDir: String + private val mappingDir: String, + private val project: Project ) { companion object { private const val TAG: String = "Matrix.Trace" @@ -68,6 +68,7 @@ class MatrixTrace( changedFiles: Map, inputToOutput: Map, isIncremental: Boolean, + skipCheckClass: Boolean, traceClassDirectoryOutput: File, legacyReplaceChangedFile: ((File, Map) -> Object)?, legacyReplaceFile: ((File, File) -> (Object))? @@ -81,6 +82,7 @@ class MatrixTrace( .setBaseMethodMap(baseMethodMapPath) .setBlockListFile(blockListFilePath) .setMappingPath(mappingDir) + .setSkipCheckClass(skipCheckClass) .build() /** @@ -152,7 +154,13 @@ class MatrixTrace( */ start = System.currentTimeMillis() val methodTracer = MethodTracer(executor, mappingCollector, config, methodCollector.collectedMethodMap, methodCollector.collectedClassExtendMap) - methodTracer.trace(dirInputOutMap, jarInputOutMap) + val allInputs = ArrayList().also { + it.addAll(dirInputOutMap.keys) + it.addAll(jarInputOutMap.keys) + } + val traceClassLoader = TraceClassLoader.getClassLoader(project, allInputs) + methodTracer.trace(dirInputOutMap, jarInputOutMap, traceClassLoader, skipCheckClass) + Log.i(TAG, "[doTransform] Step(3)[Trace]... cost:%sms", System.currentTimeMillis() - start) } diff --git a/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/transform/MatrixTraceLegacyTransform.kt b/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/transform/MatrixTraceLegacyTransform.kt index 5e41d4a01..02fe36c98 100644 --- a/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/transform/MatrixTraceLegacyTransform.kt +++ b/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/transform/MatrixTraceLegacyTransform.kt @@ -34,6 +34,7 @@ import java.util.concurrent.ConcurrentHashMap // For Android Gradle Plugin 3.5.0 class MatrixTraceLegacyTransform( + private val project: Project, private val config: Configuration, private val origTransform: Transform ) : Transform() { @@ -63,6 +64,7 @@ class MatrixTraceLegacyTransform( .setIgnoreMethodMapFilePath("$mappingOut/ignoreMethodMapping.txt") .setMappingPath(mappingOut) .setTraceClassOut(traceClassOut) + .setSkipCheckClass(extension.isSkipCheckClass) .build() val hardTask = getTransformTaskName(extension.customDexTransformName, variant.name) @@ -72,7 +74,7 @@ class MatrixTraceLegacyTransform( Log.i(TAG, "successfully inject task:" + task.name) val field = TransformTask::class.java.getDeclaredField("transform") field.isAccessible = true - field.set(task, MatrixTraceLegacyTransform(config, task.transform)) + field.set(task, MatrixTraceLegacyTransform(project, config, task.transform)) break } } @@ -170,11 +172,13 @@ class MatrixTraceLegacyTransform( methodMapFilePath = config.methodMapFilePath, baseMethodMapPath = config.baseMethodMapPath, blockListFilePath = config.blockListFilePath, - mappingDir = config.mappingDir + mappingDir = config.mappingDir, + project = project ).doTransform( classInputs = inputFiles, changedFiles = changedFiles, isIncremental = isIncremental, + skipCheckClass = config.skipCheckClass, traceClassDirectoryOutput = File(config.traceClassOut), inputToOutput = ConcurrentHashMap(), legacyReplaceChangedFile = legacyReplaceChangedFile, diff --git a/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/transform/MatrixTraceTransform.kt b/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/transform/MatrixTraceTransform.kt index 1d626b471..4c6d43907 100644 --- a/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/transform/MatrixTraceTransform.kt +++ b/matrix/matrix-android/matrix-gradle-plugin/src/main/kotlin/com/tencent/matrix/plugin/transform/MatrixTraceTransform.kt @@ -171,6 +171,7 @@ class MatrixTraceTransform( .setMethodMapFilePath("$mappingOut/methodMapping.txt") .setIgnoreMethodMapFilePath("$mappingOut/ignoreMethodMapping.txt") .setMappingPath(mappingOut) + .setSkipCheckClass(extension.isSkipCheckClass) .build() } @@ -236,11 +237,13 @@ class MatrixTraceTransform( methodMapFilePath = config.methodMapFilePath, baseMethodMapPath = config.baseMethodMapPath, blockListFilePath = config.blockListFilePath, - mappingDir = config.mappingDir + mappingDir = config.mappingDir, + project = project ).doTransform( classInputs = inputFiles, changedFiles = changedFiles, isIncremental = isIncremental, + skipCheckClass = config.skipCheckClass, traceClassDirectoryOutput = outputDirectory, inputToOutput = inputToOutput, legacyReplaceChangedFile = null, diff --git a/matrix/matrix-android/matrix-hooks/CMakeLists.txt b/matrix/matrix-android/matrix-hooks/CMakeLists.txt index 5fd216c4e..0d4509572 100644 --- a/matrix/matrix-android/matrix-hooks/CMakeLists.txt +++ b/matrix/matrix-android/matrix-hooks/CMakeLists.txt @@ -10,6 +10,7 @@ set(SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/src/main/cpp) find_library(log-lib, log) add_subdirectory(${SOURCE_DIR}/external/libcJSON) +add_subdirectory(${SOURCE_DIR}/external/fastunwind) ################################## Common Part ################################## set(TARGET matrix-hookcommon) @@ -94,3 +95,92 @@ target_link_libraries( PRIVATE -Wl,--version-script=${SOURCE_DIR}/pthread/pthread.ver ) ################################################################################# + +################################### MemGuard #################################### +set(TARGET memguard_base) + +add_library( + ${TARGET} + STATIC + ${SOURCE_DIR}/memguard/port/Hook.cpp + ${SOURCE_DIR}/memguard/port/Log.cpp + ${SOURCE_DIR}/memguard/port/Memory.cpp + ${SOURCE_DIR}/memguard/port/Mutex.cpp + ${SOURCE_DIR}/memguard/port/Paths.cpp + ${SOURCE_DIR}/memguard/port/Random.cpp + ${SOURCE_DIR}/memguard/port/Unwind.cpp + ${SOURCE_DIR}/memguard/port/FdSanWrapper.cpp + ${SOURCE_DIR}/memguard/util/SignalHandler.cpp + ${SOURCE_DIR}/memguard/util/Interception.cpp + ${SOURCE_DIR}/memguard/util/PagePool.cpp + ${SOURCE_DIR}/memguard/util/Allocation.cpp + ${SOURCE_DIR}/memguard/util/Thread.cpp + ${SOURCE_DIR}/memguard/util/Issue.cpp + ${SOURCE_DIR}/memguard/MemGuard.cpp +) + +target_include_directories( + ${TARGET} + PRIVATE ${SOURCE_DIR}/memguard + PUBLIC ${SOURCE_DIR} + PUBLIC ${EXT_DEP}/include + PUBLIC ${EXT_DEP}/include/backtrace + PUBLIC ${EXT_DEP}/include/backtrace/common +) + +target_compile_options( + ${TARGET} + PRIVATE -Wall -Wextra -Werror -Wno-unused-function + PRIVATE $<$:-std=c17> + PRIVATE $<$:-std=c++17> + PUBLIC -fvisibility=hidden -fno-exceptions -fno-rtti -fdata-sections -ffunction-sections + PUBLIC -DMEMGUARD_LOG_LEVEL=4 +) + +target_link_libraries( + ${TARGET} + PUBLIC -Wl,--gc-sections + PRIVATE ${log-lib} + PUBLIC matrix-hookcommon + PUBLIC wechatbacktrace + PUBLIC ${EXT_DEP}/lib/${ANDROID_ABI}/libunwindstack.a + PUBLIC fastunwind +) + + +set(TARGET matrix-memguard) + +add_library( + ${TARGET} + SHARED + ${SOURCE_DIR}/memguard/jni/com_tencent_mm_tools_memguard_MemGuard.cpp + ${SOURCE_DIR}/memguard/jni/com_tencent_mm_tools_memguard_MemGuard_00024Options.cpp + ${SOURCE_DIR}/memguard/jni/JNIAux.cpp + ${SOURCE_DIR}/memguard/jni/C2Java.cpp +) + +target_include_directories( + ${TARGET} + PRIVATE ${SOURCE_DIR}/memguard + PRIVATE ${EXT_DEP}/include + PRIVATE ${EXT_DEP}/include/backtrace + PRIVATE ${EXT_DEP}/include/backtrace/common +) + +target_compile_options( + ${TARGET} + PRIVATE -Wall -Wextra -Werror -Wno-unused-function + PRIVATE -fvisibility=hidden -fno-exceptions -fno-rtti -fdata-sections -ffunction-sections + PRIVATE $<$:-std=c17> + PRIVATE $<$:-std=c++17> +) + +target_link_libraries( + ${TARGET} + PRIVATE -Wl,--gc-sections + PRIVATE -Wl,--version-script=${SOURCE_DIR}/memguard/memguard.map + PRIVATE ${log-lib} + PRIVATE matrix-hookcommon + PRIVATE memguard_base +) +################################################################################# \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/common/HookCommon.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/common/HookCommon.h index 01e3da96e..51455c0c4 100644 --- a/matrix/matrix-android/matrix-hooks/src/main/cpp/common/HookCommon.h +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/common/HookCommon.h @@ -32,6 +32,7 @@ #define HOOK_REQUEST_GROUPID_DLOPEN_MON 0x02 #define HOOK_REQUEST_GROUPID_MEMORY 0x03 #define HOOK_REQUEST_GROUPID_PTHREAD 0x04 +#define HOOK_REQUEST_GROUPID_MEMGUARD 0x05 #define GET_CALLER_ADDR(__caller_addr) \ void * __caller_addr = __builtin_return_address(0) @@ -96,6 +97,7 @@ xhook_grouped_ignore(group_id, ".*/libmatrix-memoryhook\\.so$", NULL); \ xhook_grouped_ignore(group_id, ".*/libmatrix-pthreadhook\\.so$", NULL); \ xhook_grouped_ignore(group_id, ".*/libmatrix-opengl-leak\\.so$", NULL); \ + xhook_grouped_ignore(group_id, ".*/libmatrix-memguard\\.so$", NULL); \ } while (0) #include diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/common/Macros.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/common/Macros.h index 4c10331f3..16867ee4a 100644 --- a/matrix/matrix-android/matrix-hooks/src/main/cpp/common/Macros.h +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/common/Macros.h @@ -25,15 +25,11 @@ #define HOOK_LOG_ERROR(fmt, ...) //__android_log_print(ANDROID_LOG_ERROR, "TestHook", fmt, ##__VA_ARGS__) #define USE_CRITICAL_CHECK true -#define USE_MEMORY_MESSAGE_QUEUE true -#define USE_SPLAY_MAP_SAVE_STACK true -#define USE_STACK_HASH_NO_COLLISION true - -/* Incubating - currently no feasible performance(40% slower). */ -#define USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE false +#define USE_ALLOC_COUNTER true /* For testing */ -#define USE_FAKE_BACKTRACE_DATA false +#define ENABLE_FAKE_BACKTRACE_DATA false +#define ENABLE_CHECK_MESSAGE_OVERFLOW false #if USE_CRITICAL_CHECK == true #define CRITICAL_CHECK(assertion) matrix::_hook_check(assertion) @@ -42,13 +38,14 @@ #endif #define SIZE_AUGMENT 192 + +#define MEMORY_OVER_LIMIT (1024 * 1024 * 150L) // 150M + #define PROCESS_BUSY_INTERVAL 40 * 1000L #define PROCESS_NORMAL_INTERVAL 150 * 1000L #define PROCESS_LESS_NORMAL_INTERVAL 300 * 1000L #define PROCESS_IDLE_INTERVAL 800 * 1000L -#define MEMORY_OVER_LIMIT 1024 * 1024 * 200L // 200M - #define PTR_SPLAY_MAP_CAPACITY 10240 #define STACK_SPLAY_MAP_CAPACITY 1024 @@ -56,9 +53,13 @@ #define POINTER_MASK 48 +#ifndef LIKELY #define LIKELY(cond) (__builtin_expect(!!(cond), 1)) +#endif +#ifndef UNLIKELY #define UNLIKELY(cond) (__builtin_expect(!!(cond), 0)) +#endif #define EXPORT extern __attribute__ ((visibility ("default"))) @@ -71,5 +72,12 @@ abort(); \ } +#if ENABLE_CHECK_MESSAGE_OVERFLOW == true +#define CHECK_MESSAGE_OVERFLOW(assertion) HOOK_CHECK(assertion) +#else +#define CHECK_MESSAGE_OVERFLOW(assertion) +#endif + +#define ReservedSize(AugmentExp) ((1 << AugmentExp)) #endif //LIBMATRIX_JNI_MACROS_H diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/common/SoLoadMonitor.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/common/SoLoadMonitor.h index 1dacd19ac..4f5bf76dd 100644 --- a/matrix/matrix-android/matrix-hooks/src/main/cpp/common/SoLoadMonitor.h +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/common/SoLoadMonitor.h @@ -6,10 +6,12 @@ #define MATRIX_ANDROID_SOLOADMONITOR_H +#include "Macros.h" + namespace matrix { typedef void (*so_load_callback_t)(const char *__file_name); - extern bool InstallSoLoadMonitor(); + EXPORT bool InstallSoLoadMonitor(); EXPORT void AddOnSoLoadCallback(so_load_callback_t cb); EXPORT void PauseLoadSo(); EXPORT void ResumeLoadSo(); diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/common/struct/buffer_source.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/common/struct/buffer_source.h index 1d55029b2..3e3bb2841 100644 --- a/matrix/matrix-android/matrix-hooks/src/main/cpp/common/struct/buffer_source.h +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/common/struct/buffer_source.h @@ -54,11 +54,15 @@ class buffer_source_memory : public buffer_source { static std::atomic g_realloc_memory_counter; virtual void *realloc(size_t new_size) { - if (_buffer) g_realloc_counter.fetch_add(1); + + if (_buffer) g_realloc_counter.fetch_add(1, std::memory_order_relaxed); + void *ptr = ::realloc(_buffer, new_size); if (ptr != NULL) { - g_realloc_memory_counter.fetch_sub(_buffer_size); - g_realloc_memory_counter.fetch_add(new_size); + + g_realloc_memory_counter.fetch_sub(_buffer_size, std::memory_order_relaxed); + g_realloc_memory_counter.fetch_add(new_size, std::memory_order_relaxed); + _buffer = ptr; _buffer_size = new_size; } @@ -68,7 +72,9 @@ class buffer_source_memory : public buffer_source { virtual void free() { if (_buffer) { - g_realloc_memory_counter.fetch_sub(_buffer_size); + + g_realloc_memory_counter.fetch_sub(_buffer_size, std::memory_order_relaxed); + ::free(_buffer); _buffer = NULL; _buffer_size = 0; diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/common/struct/lock_free_array_queue.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/common/struct/lock_free_array_queue.h new file mode 100644 index 000000000..bf791e201 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/common/struct/lock_free_array_queue.h @@ -0,0 +1,202 @@ +/* + * Tencent is pleased to support the open source community by making wechat-matrix available. + * Copyright (C) 2019 THL A29 Limited, a Tencent company. All rights reserved. + * Licensed under the BSD 3-Clause License (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://opensource.org/licenses/BSD-3-Clause + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef lock_free_array_queue_h +#define lock_free_array_queue_h + +#include +#include + +#include + +namespace matrix { + +#define CACHE_LINE_SIZE 64 + + template + class ObjectStorage { + public: + ObjectStorage() = default; + virtual ~ObjectStorage() = default; + virtual size_t provide() = 0; + virtual T *get(size_t idx) = 0; + }; + + template + class FixedObjectStorage : public ObjectStorage { + public: + FixedObjectStorage() = default; + + virtual ~FixedObjectStorage() = default; + + inline size_t provide() override { + size_t idx = available_.fetch_add(1); + if (UNLIKELY(idx >= Reserved)) { + available_.store(Reserved); + return SIZE_MAX; + } + return idx; + } + + inline T *get(size_t idx) override { + return &objects[idx]; + } + + std::atomic available_ = 0; + + private: + T objects[Reserved]; + }; + + template + class SPMC_FixedFreeList { + public: + SPMC_FixedFreeList(): mask_(Reserved - 1), head_(Reserved), tail_(0) { + for (size_t i = 0; i < Reserved; i++) { + free_[i] = i; + } + }; + + virtual ~SPMC_FixedFreeList() = default; + + uint32_t allocate_spmc() { + + size_t idx; + for (;;) { + size_t head = head_.load(std::memory_order_acquire); + size_t tail = tail_.load(std::memory_order_acquire); + if (UNLIKELY(tail == head)) { + return INT_MAX; + } + + idx = free_[tail & mask_]; + if (tail_.compare_exchange_weak(tail, tail + 1)) { + break; + } + } + return idx; + } + + void deallocate_spmc(uint32_t idx) { + + size_t head = head_.load(std::memory_order_acquire); + free_[head & mask_] = idx; + bool ok = head_.compare_exchange_strong(head, head + 1); + HOOK_CHECK(ok); + return; + } + + T * get(size_t idx) { + return storage_.get(idx); + } + + private: + + const size_t mask_; + + FixedObjectStorage storage_; + + char pad1[CACHE_LINE_SIZE - sizeof(std::atomic)]; + std::atomic head_; + char pad2[CACHE_LINE_SIZE - sizeof(std::atomic)]; + std::atomic tail_; + char pad3[CACHE_LINE_SIZE - sizeof(std::atomic)]; + + uint32_t free_[Reserved]; + }; + + template + class LockFreeArrayQueue { + public: + LockFreeArrayQueue() : mask_(Reserved - 1), size_(Reserved), head_first_(0), head_second_(0), tail_first_(0) { + } + + ~LockFreeArrayQueue() = default; + + bool offer_mpsc(const T & t) { + + uint32_t head, tail, next; + + for (;;) { + head = head_first_.load(std::memory_order_acquire); + tail = tail_first_.load(std::memory_order_acquire); + if (UNLIKELY((head - tail) > mask_)) { + return false; + } + next = head + 1; + if (head_first_.compare_exchange_weak(head, next)) { + break; + } + } + + messages_[head & mask_] = t; + + for (;;) { + if (head_second_.compare_exchange_weak(head, next)) { + break; + } else { + pause(); + } + } + + return true; + } + + bool poll_mpsc(T & ret) { + + uint32_t tail, head; + int ok; + + tail = tail_first_.load(std::memory_order_acquire); + head = head_second_.load(std::memory_order_acquire); + + if (tail == head) + return false; + + HOOK_CHECK(!(tail > head && (head - tail) > mask_)); + + ret = messages_[tail & mask_]; + ok = tail_first_.compare_exchange_strong(tail, tail + 1); + HOOK_CHECK(ok) + + return true; + } + + void pause() { +// sched_yield(); +// sleep(0); + }; + + private: + const uint32_t mask_; + const uint32_t size_; + + char pad0[CACHE_LINE_SIZE - 2 * sizeof(uint32_t)]; + + std::atomic head_first_; + char pad1[CACHE_LINE_SIZE - sizeof(std::atomic)]; + std::atomic head_second_; + char pad2[CACHE_LINE_SIZE - sizeof(std::atomic)]; + std::atomic tail_first_; + char pad3[CACHE_LINE_SIZE - sizeof(std::atomic)]; + + T messages_[Reserved]; + }; + + + +} +#endif /* lock_free_array_queue_h */ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/common/struct/lock_free_queue.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/common/struct/lock_free_queue.h index 2128eba26..2f8dbe8cb 100644 --- a/matrix/matrix-android/matrix-hooks/src/main/cpp/common/struct/lock_free_queue.h +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/common/struct/lock_free_queue.h @@ -341,36 +341,6 @@ namespace matrix { } } - /* - bool poll_mpsc(T & ret) { - - auto head = head_.load(std::memory_order_acquire); - auto head_node = head.ptr_; - - for (;;) { - - auto tail = tail_.load(std::memory_order_acquire); - - auto next = head_node->next_.load(std::memory_order_acquire); - auto next_node = next.ptr_; - - if (head == tail) { - if (next_node == nullptr) - return false; - TaggedPointer> new_tail(next_node, tail.tag_ + 1); - tail_.compare_exchange_strong(tail, new_tail); - } else { - ret = next_node->t_; - TaggedPointer> new_head(next_node, head.tag_ + 1); - head_.store(new_head); - free_list_->deallocate(head_node); - return true; - } - - } - } - */ - private: std::atomic>> head_; diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/external/fastunwind/CMakeLists.txt b/matrix/matrix-android/matrix-hooks/src/main/cpp/external/fastunwind/CMakeLists.txt new file mode 100644 index 000000000..5861a8539 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/external/fastunwind/CMakeLists.txt @@ -0,0 +1,17 @@ +cmake_minimum_required(VERSION 3.6) + +add_library( + fastunwind + STATIC + ${CMAKE_CURRENT_SOURCE_DIR}/FastUnwind.cpp +) + +target_compile_options( + fastunwind + PRIVATE -Wall -Wextra -Werror -fvisibility=hidden +) + +target_include_directories( + fastunwind + PUBLIC ${CMAKE_CURRENT_SOURCE_DIR} +) \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/external/fastunwind/FastUnwind.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/external/fastunwind/FastUnwind.cpp new file mode 100644 index 000000000..a1481b8e7 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/external/fastunwind/FastUnwind.cpp @@ -0,0 +1,130 @@ +// +// Created by tomystang on 2020/12/15. +// + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "FastUnwind.h" + +using namespace fastunwind; + +#define LIKELY(cond) __builtin_expect(!!(cond), 1) +#define UNLIKELY(cond) __builtin_expect(!!(cond), 0) +#define TLSVAR __thread __attribute__((tls_model("initial-exec"))) + +#if 0 // defined(__aarch64__) arm 8.1a feature, disable so far to avoid compatibility issues in ndk r21. +#define HWCAP2_MTE (1 << 18) + +struct ScopedDisableMTE { + size_t prevTCO = {}; + + ScopedDisableMTE() { + if (IsMTESupported()) { + __asm__ __volatile__(".arch_extension mte; mrs %0, tco; msr tco, #1" : "=r"(prevTCO)); + } + } + + ~ScopedDisableMTE() { + if (IsMTESupported()) { + __asm__ __volatile__(".arch_extension mte; msr tco, %0" : : "r"(prevTCO)); + } + } + + static inline bool IsMTESupported() { + return (::getauxval(AT_HWCAP2) & HWCAP2_MTE) != 0; + } +}; +#else +struct ScopedDisableMTE { + ScopedDisableMTE() = default; +}; +#endif + +static uintptr_t TLSVAR sThreadStackTop = 0; + +static inline __attribute__((unused)) uintptr_t GetThreadStackTop() { + uintptr_t threadStackTop = sThreadStackTop; + if (UNLIKELY(threadStackTop == 0)) { + pthread_attr_t attr = {}; + if (pthread_getattr_np(pthread_self(), &attr) != 0) { + return 0; + } + void* stackBase = nullptr; + size_t stackSize = 0; + pthread_attr_getstack(&attr, &stackBase, &stackSize); + threadStackTop = (uintptr_t) stackBase + stackSize; + sThreadStackTop = threadStackTop; + } + return threadStackTop; +} + +static inline __attribute__((unused)) uintptr_t ClearPACBits(uintptr_t ptr) { +#if defined(__aarch64__) + register uintptr_t x30 __asm("x30") = ptr; + // This is a NOP on pre-Armv8.3-A architectures. + asm("xpaclri" : "+r"(x30)); + return x30; +#else + return ptr; +#endif +} + +static int UnwindImpl(const void* begin_fp, void** pcs, size_t max_count) { + ScopedDisableMTE __attribute__((unused)) x; + + auto begin = reinterpret_cast(begin_fp); + uintptr_t end = GetThreadStackTop(); + + stack_t ss = {}; + if (UNLIKELY(::sigaltstack(nullptr, &ss) == 0 && (ss.ss_flags & SS_ONSTACK) != 0)) { + end = (uintptr_t) ss.ss_sp + ss.ss_size; + } + + struct FrameRecord { + uintptr_t nextFrame; + uintptr_t returnAddr; + }; + + size_t numFrames = 0; + while (true) { + auto* frame = (FrameRecord*) begin; + if (frame->nextFrame < begin + sizeof(FrameRecord) + || frame->nextFrame >= end + || frame->nextFrame % sizeof(void*) != 0) { + break; + } + if (numFrames < max_count) { + pcs[numFrames++] = (void*) ClearPACBits(frame->returnAddr); + } else { + break; + } + begin = frame->nextFrame; + } + return numFrames; +} + +int fastunwind::Unwind(void** pcs, size_t max_count) { + return UnwindImpl(__builtin_frame_address(0), pcs, max_count); +} + +int fastunwind::Unwind(void* ucontext, void** pcs, size_t max_count) { +#if defined (__aarch64__) + const void* begin_fp = *reinterpret_cast(reinterpret_cast(ucontext) + 432); + return UnwindImpl(begin_fp, pcs, max_count); +#elif defined (__i386__) + const void* begin_fp = *reinterpret_cast(reinterpret_cast(ucontext) + 48); + return UnwindImpl(begin_fp, pcs, max_count); +#else + (void) (ucontext); + (void) (pcs); + (void) (max_count); + return 0; +#endif +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/external/fastunwind/FastUnwind.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/external/fastunwind/FastUnwind.h new file mode 100644 index 000000000..463afa87f --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/external/fastunwind/FastUnwind.h @@ -0,0 +1,16 @@ +// +// Created by tomystang on 2020/12/15. +// + +#ifndef __MEMGUARD_FASTUNWIND_H__ +#define __MEMGUARD_FASTUNWIND_H__ + +#include + +namespace fastunwind { + extern __attribute__((no_sanitize("address", "hwaddress"))) int Unwind(void** pcs, size_t max_count); + extern __attribute__((no_sanitize("address", "hwaddress"))) int Unwind(void* ucontext, void** pcs, size_t max_count); +} + + +#endif //__MEMGUARD_FASTUNWIND_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/MemGuard.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/MemGuard.cpp new file mode 100644 index 000000000..e3c85f0d5 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/MemGuard.cpp @@ -0,0 +1,76 @@ +// +// Created by tomystang on 2020/11/6. +// + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "MemGuard.h" + +using namespace memguard; + +#define LOG_TAG "MemGuard.Native" + +memguard::Options memguard::gOpts; +static std::atomic_flag sInstalled(false); + +bool memguard::Install(const Options* opts) { + if (opts == nullptr) { + LOGE(LOG_TAG, "opts == nullptr"); + return false; + } + + if (sInstalled.test_and_set()) { + LOGW(LOG_TAG, "Already installed."); + return true; + } + + gOpts = *opts; + + if (!paths::Exists(gOpts.issueDumpFilePath)) { + if (!paths::MakeDirs(paths::GetParent(gOpts.issueDumpFilePath))) { + LOGE(LOG_TAG, "Fail to create directory for issue dump file: %s", gOpts.issueDumpFilePath.c_str()); + return false; + } + } + + if (!matrix::InstallSoLoadMonitor()) { + LOGE(LOG_TAG, "Fail to install SoLoadMonitor."); + return false; + } + + if (!pagepool::Prepare()) { + LOGE(LOG_TAG, "Fail to prepare page pool."); + return false; + } + + if (!unwind::Initialize()) { + LOGE(LOG_TAG, "Fail to initialize stack unwinder."); + return false; + } + + if (!signalhandler::Install()) { + LOGE(LOG_TAG, "Fail to install signal handler."); + return false; + } + + if (!allocation::Prepare()) { + LOGE(LOG_TAG, "Fail to prepare allocation logic."); + return false; + } + + if (!interception::Install()) { + LOGE(LOG_TAG, "Fail to install interceptors."); + return false; + } + + LOGI(LOG_TAG, "MemGuard was installed."); + return true; +} diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/MemGuard.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/MemGuard.h new file mode 100644 index 000000000..cb9ab7e49 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/MemGuard.h @@ -0,0 +1,18 @@ +// +// Created by tomystang on 2020/11/6. +// + +#ifndef __MEMGUARD_MEMGUARD_H__ +#define __MEMGUARD_MEMGUARD_H__ + + +#include +#include +#include "Options.h" + +namespace memguard { + extern bool Install(const Options* opts); +} + + +#endif //__MEMGUARD_MEMGUARD_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/Options.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/Options.h new file mode 100644 index 000000000..809b089c2 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/Options.h @@ -0,0 +1,59 @@ +// +// Created by tomystang on 2020/10/16. +// + +#ifndef __MEMGUARD_OPTIONS_H__ +#define __MEMGUARD_OPTIONS_H__ + + +#include +#include +#include +#include +#include +#include + +namespace memguard { + struct Options { + #define OPTION_ITEM(type, name, default_value, description) type name; + #include "Options.inc" + #undef OPTION_ITEM + + Options() { + #define OPTION_ITEM(type, name, default_value, description) name = default_value; + #include "Options.inc" + #undef OPTION_ITEM + } + + Options(const Options& other) { + #define OPTION_ITEM(type, name, default_value, description) this->name = other.name; + #include "Options.inc" + #undef OPTION_ITEM + } + + Options(Options&& other) { + #define OPTION_ITEM(type, name, default_value, description) this->name = std::move(other.name); + #include "Options.inc" + #undef OPTION_ITEM + } + + Options& operator =(const Options& rhs) { + #define OPTION_ITEM(type, name, default_value, description) this->name = rhs.name; + #include "Options.inc" + #undef OPTION_ITEM + return *this; + } + + Options& operator =(Options&& rhs) { + #define OPTION_ITEM(type, name, default_value, description) this->name = std::move(rhs.name); + #include "Options.inc" + #undef OPTION_ITEM + return *this; + } + }; + + extern Options gOpts; +} + + +#endif //__MEMGUARD_OPTIONS_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/Options.inc b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/Options.inc new file mode 100644 index 000000000..6f5b0233f --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/Options.inc @@ -0,0 +1,42 @@ +// +// Created by tomystang on 2020/10/18. +// + +OPTION_ITEM(size_t, maxAllocationSize, 64 * 1024, + "Max allocation size MemGuard can detect its under/overflow " + "issues.") + +OPTION_ITEM(size_t, maxDetectableAllocationCount, 16384, + "Max allocation count MemGuard can detect its under/overflow " + "issues.") + +OPTION_ITEM(size_t, maxSkippedAllocationCount, 5, + "Max skipped allocation count between two guarded allocations. " + "For example, if 5 was set to this option, MemGuard will generate a " + "random number 'k' in range [0,5] and the first k-th allocations " + "will be ignored.") + +OPTION_ITEM(size_t, percentageOfLeftSideGuard, 30, + "Probability of putting guard page on the left side of specific pointer. " + "For example, if 30 was set to this option, the probability of a pointer being " + "guarded on the left side will be 30%, and the probability of a pointer being guarded " + "on the right side will be 70%.") + +OPTION_ITEM(bool, perfectRightSideGuard, false, + "Whether MemGuard should return a pointer with guard page on right side without " + "gaps. If true was set to this option, overflow issue will be easier to be detected " + "but the returned pointer may not be aligned properly. Sometimes these not aligned " + "pointers can crash your app.") + +OPTION_ITEM(bool, ignoreOverlappedReading, false, + "Whether MemGuard should regard overlapped reading as an issue.") + +OPTION_ITEM(std::string, issueDumpFilePath, "", + "Path to write dump file when memory issue was detected. Leave it empty " + "will make MemGuard not dump issue info into file.") + +OPTION_ITEM(std::vector, targetSOPatterns, std::vector(), + "Patterns described by RegEx of target libs that we want to detect any memory issues.") + +OPTION_ITEM(std::vector, ignoredSOPatterns, std::vector(), + "Patterns described by RegEx of target libs that we want to skip for detecting any memory issues.") diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/C2Java.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/C2Java.cpp new file mode 100644 index 000000000..895b468b1 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/C2Java.cpp @@ -0,0 +1,47 @@ +// +// Created by tomystang on 2021/2/1. +// + +#include +#include +#include +#include "JNIAux.h" +#include "C2Java.h" + +using namespace memguard; + +#define LOG_TAG "MemGuard.C2JAVA" + +static jclass sMemGuardClazz = nullptr; +static jmethodID sMethodID_MemGuard_c2jNotifyOnIssueDumpped = nullptr; +static std::atomic_flag sInstalled(false); + +#define PREPARE_METHODID(env, var, clazz, name, sig, is_static, failure_ret_val) \ + do { \ + if (!(var)) { \ + if (is_static) { \ + (var) = (env)->GetStaticMethodID(clazz, name, sig); \ + } else { \ + (var) = (env)->GetMethodID(clazz, name, sig); \ + } \ + RETURN_ON_EXCEPTION(env, failure_ret_val); \ + } \ + } while (false) + +bool memguard::c2j::Prepare(jclass memguard_clazz) { + if (sInstalled.test_and_set()) { + LOGW(LOG_TAG, "Already prepared."); + return true; + } + auto env = jni::GetEnv(); + sMemGuardClazz = (jclass) env->NewGlobalRef(memguard_clazz); + PREPARE_METHODID(env, sMethodID_MemGuard_c2jNotifyOnIssueDumpped, memguard_clazz, + "c2jNotifyOnIssueDumped", "(Ljava/lang/String;)V", true, false); + return true; +} + +void memguard::c2j::NotifyOnIssueDumpped(const char *dump_file) { + auto env = jni::GetEnv(); + auto jDumpFile = env->NewStringUTF(dump_file); + env->CallStaticVoidMethod(sMemGuardClazz, sMethodID_MemGuard_c2jNotifyOnIssueDumpped, jDumpFile); +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/C2Java.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/C2Java.h new file mode 100644 index 000000000..825d369c7 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/C2Java.h @@ -0,0 +1,19 @@ +// +// Created by tomystang on 2021/2/1. +// + +#ifndef __MEMGUARD_C2JAVA_H__ +#define __MEMGUARD_C2JAVA_H__ + + +#include + +namespace memguard { + namespace c2j { + extern bool Prepare(jclass memguard_clazz); + extern void NotifyOnIssueDumpped(const char* dump_file); + } +} + + +#endif //__MEMGUARD_C2JAVA_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/JNIAux.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/JNIAux.cpp new file mode 100644 index 000000000..c0036f83e --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/JNIAux.cpp @@ -0,0 +1,60 @@ +// +// Created by tomystang on 2020/11/18. +// + +#include +#include +#include +#include +#include "JNIAux.h" + +using namespace memguard; + +#define LOG_TAG "MemGuard.JNIAux" + +static JavaVM* sVM; +static pthread_once_t sInitOnceKey; +static pthread_key_t sAttachFlagTLSKey; + +static jclass sClass_Throwable = nullptr; +static jmethodID sMethodID_printStackTrace = nullptr; + +bool memguard::jni::InitializeEnv(JavaVM* vm) { + sVM = vm; + + auto env = GetEnv(); + sClass_Throwable = (jclass) env->NewGlobalRef(env->FindClass("java/lang/Throwable")); + RETURN_ON_EXCEPTION(env, false); + sMethodID_printStackTrace = env->GetMethodID(sClass_Throwable, "printStackTrace", "()V"); + RETURN_ON_EXCEPTION(env, false); + + LOGD(LOG_TAG, "InitializeEnv done."); + return true; +} + +JNIEnv* memguard::jni::GetEnv() { + ASSERT(sVM != nullptr, "Not initialized."); + JNIEnv* result = nullptr; + auto ret = sVM->GetEnv((void**) &result, JNI_VERSION_1_6); + ASSERT(ret != JNI_EVERSION, "System is too old to be supported."); + if (ret != JNI_OK) { + pthread_once(&sInitOnceKey, []() { + pthread_key_create(&sAttachFlagTLSKey, [](void* flag) { + if (flag) { + sVM->DetachCurrentThread(); + } + }); + }); + if (sVM->AttachCurrentThread(&result, nullptr) == JNI_OK) { + pthread_setspecific(sAttachFlagTLSKey, (void*) 1); + } else { + result = nullptr; + } + } + ASSERT(result != nullptr, "Fail to get JNIEnv on current thread (%d).", gettid()); + return result; +} + +void memguard::jni::PrintStackTrace(JNIEnv *env, jthrowable thr) { + env->CallVoidMethod(thr, sMethodID_printStackTrace); +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/JNIAux.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/JNIAux.h new file mode 100644 index 000000000..7eeab7e58 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/JNIAux.h @@ -0,0 +1,48 @@ +// +// Created by tomystang on 2020/11/18. +// + +#ifndef __MEMGUARD_JNIAUX_H__ +#define __MEMGUARD_JNIAUX_H__ + + +#include +#include +#include +#include +#include + +namespace memguard { + namespace jni { + extern bool InitializeEnv(JavaVM* vm); + extern JNIEnv* GetEnv(); + + template ::value, void>::type> + ScopedObject> MakeScopedLocalRef(TRef ref) { + auto dtor_lambda = [](TRef ref) { + auto env = GetEnv(); + if (ref != nullptr && env->GetObjectRefType(ref) == JNILocalRefType) { + env->DeleteLocalRef(ref); + } + }; + return MakeScopedObject(std::forward(ref), std::function(dtor_lambda)); + } + + extern void PrintStackTrace(JNIEnv* env, jthrowable thr); + } +} + +#define RETURN_ON_EXCEPTION(env, ret_value) \ + do { \ + if ((env)->ExceptionCheck()) { \ + return (ret_value); \ + } \ + } while (false) + +#define JNI_METHOD(name) __JNI_METHOD_1(JNI_CLASS_PREFIX, name) +#define __JNI_METHOD_1(prefix, name) __JNI_METHOD_2(prefix, name) +#define __JNI_METHOD_2(prefix, name) JNIEXPORT Java_##prefix##_##name + + +#endif //__MEMGUARD_JNIAUX_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/com_tencent_mm_tools_memguard_MemGuard.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/com_tencent_mm_tools_memguard_MemGuard.cpp new file mode 100644 index 000000000..ae86ecdde --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/com_tencent_mm_tools_memguard_MemGuard.cpp @@ -0,0 +1,48 @@ +// +// Created by tomystang on 2020/10/15. +// + +#include +#include +#include +#include +#include +#include +#include +#include +#include "JNIAux.h" +#include "com_tencent_mm_tools_memguard_MemGuard_00024Options.h" +#include "C2Java.h" + +using namespace memguard; + +#define LOG_TAG "MemGuard.JNI" + +#define JNI_CLASS_PREFIX com_tencent_matrix_memguard_MemGuard + +extern "C" jint JNI_OnLoad(JavaVM* vm, void*) { + if (!jni::InitializeEnv(vm)) { + LOGE(LOG_TAG, "Fail to initialize jni environment."); + return JNI_ERR; + } + LOGI(LOG_TAG, "Libmemguard was loaded."); + return JNI_VERSION_1_6; +} + +extern "C" jboolean JNI_METHOD(nativeInstall)(JNIEnv* env, jclass memguard_clazz, jobject opts) { + Options raw_opts; + jni::FillOptWithJavaOptions(env, opts, &raw_opts); + c2j::Prepare(memguard_clazz); + if (Install(&raw_opts)) { + return JNI_TRUE; + } else { + return JNI_FALSE; + } +} + +extern "C" jstring JNI_METHOD(nativeGetIssueDumpFilePath)(JNIEnv* env, jclass) { + if (gOpts.issueDumpFilePath.empty()) { + return nullptr; + } + return env->NewStringUTF(gOpts.issueDumpFilePath.c_str()); +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/com_tencent_mm_tools_memguard_MemGuard_00024Options.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/com_tencent_mm_tools_memguard_MemGuard_00024Options.cpp new file mode 100644 index 000000000..12f46e049 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/com_tencent_mm_tools_memguard_MemGuard_00024Options.cpp @@ -0,0 +1,126 @@ +// +// Created by tomystang on 2020/10/15. +// + +#include +#include +#include +#include +#include +#include +#include "com_tencent_mm_tools_memguard_MemGuard_00024Options.h" + +using namespace memguard; + +static jfieldID sFieldID_MemGuard$Options_maxAllocationSize = nullptr; +static jfieldID sFieldID_MemGuard$Options_maxDetectableAllocationCount = nullptr; +static jfieldID sFieldID_MemGuard$Options_maxSkippedAllocationCount = nullptr; +static jfieldID sFieldID_MemGuard$Options_percentageOfLeftSideGuard = nullptr; +static jfieldID sFieldID_MemGuard$Options_perfectRightSideGuard = nullptr; +static jfieldID sFieldID_MemGuard$Options_ignoreOverlappedReading = nullptr; +static jfieldID sFieldID_MemGuard$Options_issueDumpFilePath = nullptr; +static jfieldID sFieldID_MemGuard$Options_targetSOPatterns = nullptr; +static jfieldID sFieldID_MemGuard$Options_ignoredSOPatterns = nullptr; + +#define PREPARE_FIELDID(env, var, clazz, name, sig, failure_ret_val) \ + do { \ + if (!(var)) { \ + (var) = (env)->GetFieldID(clazz, name, sig); \ + RETURN_ON_EXCEPTION(env, failure_ret_val); \ + } \ + } while (false) + +bool memguard::jni::FillOptWithJavaOptions(JNIEnv *env, jobject jopts, Options* opts) { + ASSERT(jopts != nullptr, "jopts is null."); + ASSERT(opts != nullptr, "opts is null."); + + auto jopts_class = MakeScopedLocalRef(env->GetObjectClass(jopts)); + RETURN_ON_EXCEPTION(env, false); + + PREPARE_FIELDID(env, sFieldID_MemGuard$Options_maxAllocationSize, jopts_class.get(), + "maxAllocationSize", "I", false); + opts->maxAllocationSize = env->GetIntField(jopts, sFieldID_MemGuard$Options_maxAllocationSize); + RETURN_ON_EXCEPTION(env, false); + + PREPARE_FIELDID(env, sFieldID_MemGuard$Options_maxDetectableAllocationCount, jopts_class.get(), + "maxDetectableAllocationCount", "I", false); + opts->maxDetectableAllocationCount = env->GetIntField(jopts, sFieldID_MemGuard$Options_maxDetectableAllocationCount); + RETURN_ON_EXCEPTION(env, false); + + PREPARE_FIELDID(env, sFieldID_MemGuard$Options_maxSkippedAllocationCount, jopts_class.get(), + "maxSkippedAllocationCount", "I", false); + opts->maxSkippedAllocationCount = env->GetIntField(jopts, sFieldID_MemGuard$Options_maxSkippedAllocationCount); + RETURN_ON_EXCEPTION(env, false); + + PREPARE_FIELDID(env, sFieldID_MemGuard$Options_percentageOfLeftSideGuard, jopts_class.get(), + "percentageOfLeftSideGuard", "I", false); + opts->percentageOfLeftSideGuard = env->GetIntField(jopts, sFieldID_MemGuard$Options_percentageOfLeftSideGuard); + RETURN_ON_EXCEPTION(env, false); + + PREPARE_FIELDID(env, sFieldID_MemGuard$Options_perfectRightSideGuard, jopts_class.get(), + "perfectRightSideGuard", "Z", false); + opts->perfectRightSideGuard = env->GetBooleanField(jopts, sFieldID_MemGuard$Options_perfectRightSideGuard); + RETURN_ON_EXCEPTION(env, false); + + PREPARE_FIELDID(env, sFieldID_MemGuard$Options_ignoreOverlappedReading, jopts_class.get(), + "ignoreOverlappedReading", "Z", false); + opts->ignoreOverlappedReading = env->GetBooleanField(jopts, sFieldID_MemGuard$Options_ignoreOverlappedReading); + RETURN_ON_EXCEPTION(env, false); + + PREPARE_FIELDID(env, sFieldID_MemGuard$Options_issueDumpFilePath, jopts_class.get(), + "issueDumpFilePath", "Ljava/lang/String;", false); + auto jIssueDumpFilePath = MakeScopedLocalRef( + (jstring) env->GetObjectField(jopts, sFieldID_MemGuard$Options_issueDumpFilePath)); + RETURN_ON_EXCEPTION(env, false); + if (jIssueDumpFilePath.get() != nullptr) { + auto cIssueDumpFilePath = env->GetStringUTFChars(jIssueDumpFilePath.get(), nullptr); + if (cIssueDumpFilePath != nullptr) { + opts->issueDumpFilePath = cIssueDumpFilePath; + } + env->ReleaseStringUTFChars(jIssueDumpFilePath.get(), cIssueDumpFilePath); + } + + PREPARE_FIELDID(env, sFieldID_MemGuard$Options_targetSOPatterns, jopts_class.get(), + "targetSOPatterns", "[Ljava/lang/String;", false); + auto j_target_so_patterns_array = MakeScopedLocalRef( + (jobjectArray) env->GetObjectField(jopts, sFieldID_MemGuard$Options_targetSOPatterns)); + RETURN_ON_EXCEPTION(env, false); + if (j_target_so_patterns_array.get() != nullptr) { + jint pattern_count = env->GetArrayLength(j_target_so_patterns_array.get()); + for (int i = 0; i < pattern_count; ++i) { + auto j_pattern = MakeScopedLocalRef((jstring) env->GetObjectArrayElement( + j_target_so_patterns_array.get(), i)); + RETURN_ON_EXCEPTION(env, false); + if (j_pattern.get() == nullptr) { + continue; + } + jsize mutf_len = env->GetStringUTFLength(j_pattern.get()); + const char* mutf_pattern = env->GetStringUTFChars(j_pattern.get(), nullptr); + opts->targetSOPatterns.emplace_back(std::string(mutf_pattern, mutf_len)); + env->ReleaseStringUTFChars(j_pattern.get(), mutf_pattern); + } + } + + PREPARE_FIELDID(env, sFieldID_MemGuard$Options_ignoredSOPatterns, jopts_class.get(), + "ignoredSOPatterns", "[Ljava/lang/String;", false); + auto j_ignored_so_patterns_array = MakeScopedLocalRef( + (jobjectArray) env->GetObjectField(jopts, sFieldID_MemGuard$Options_ignoredSOPatterns)); + RETURN_ON_EXCEPTION(env, false); + if (j_ignored_so_patterns_array.get() != nullptr) { + jint pattern_count = env->GetArrayLength(j_ignored_so_patterns_array.get()); + for (int i = 0; i < pattern_count; ++i) { + auto j_pattern = MakeScopedLocalRef((jstring) env->GetObjectArrayElement( + j_ignored_so_patterns_array.get(), i)); + RETURN_ON_EXCEPTION(env, false); + if (j_pattern.get() == nullptr) { + continue; + } + jsize mutf_len = env->GetStringUTFLength(j_pattern.get()); + const char* mutf_pattern = env->GetStringUTFChars(j_pattern.get(), nullptr); + opts->ignoredSOPatterns.emplace_back(std::string(mutf_pattern, mutf_len)); + env->ReleaseStringUTFChars(j_pattern.get(), mutf_pattern); + } + } + + return true; +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/com_tencent_mm_tools_memguard_MemGuard_00024Options.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/com_tencent_mm_tools_memguard_MemGuard_00024Options.h new file mode 100644 index 000000000..9a11fc159 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/jni/com_tencent_mm_tools_memguard_MemGuard_00024Options.h @@ -0,0 +1,18 @@ +// +// Created by tomystang on 2020/11/19. +// + +#ifndef __MEMGUARD_COM_TENCENT_MM_TOOLS_MEMGUARD_MEMGUARD_00024OPTIONS_H__ +#define __MEMGUARD_COM_TENCENT_MM_TOOLS_MEMGUARD_MEMGUARD_00024OPTIONS_H__ + + +#include + +namespace memguard { + namespace jni { + extern bool FillOptWithJavaOptions(JNIEnv *env, jobject jopts, Options* opts); + } +} + + +#endif //__MEMGUARD_COM_TENCENT_MM_TOOLS_MEMGUARD_MEMGUARD_00024OPTIONS_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/memguard.map b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/memguard.map new file mode 100644 index 000000000..ddea0d383 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/memguard.map @@ -0,0 +1,7 @@ +memguard { + global: + JNI_OnLoad; + Java_*; + local: + *; +}; \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/FdSanWrapper.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/FdSanWrapper.cpp new file mode 100644 index 000000000..62d3317f4 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/FdSanWrapper.cpp @@ -0,0 +1,23 @@ +// +// Created by YinSheng Tang on 2021/9/22. +// + +#include "util/FdSanWrapper.h" + +extern "C" enum android_fdsan_error_level android_fdsan_get_error_level() __attribute__((__weak__)); + +extern "C" enum android_fdsan_error_level android_fdsan_set_error_level(enum android_fdsan_error_level new_level) __attribute__((__weak__)); + +android_fdsan_error_level AndroidFdSanGetErrorLevel() { + if (!android_fdsan_get_error_level) { + return android_fdsan_error_level::ANDROID_FDSAN_ERROR_LEVEL_DISABLED; + } + return android_fdsan_get_error_level(); +} + +android_fdsan_error_level AndroidFdSanSetErrorLevel(enum android_fdsan_error_level new_level) { + if (!android_fdsan_set_error_level) { + return android_fdsan_error_level::ANDROID_FDSAN_ERROR_LEVEL_DISABLED; + } + return android_fdsan_set_error_level(new_level); +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Hook.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Hook.cpp new file mode 100644 index 000000000..cf5deac03 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Hook.cpp @@ -0,0 +1,75 @@ +// +// Created by tomystang on 2020/10/15. +// + +#include +#include +#include +#include +#include +#include + +using namespace memguard; + +#define LOG_TAG "MemGuard.Hook" + +bool memguard::BeginHook() { + LOGD(LOG_TAG, "BeginHook()"); + return true; +} + +bool memguard::DoHook(const char* pathname_regex, const char* sym_name, void* handler_func, void** original_func) { + int ret = xhook_grouped_register(HOOK_REQUEST_GROUPID_MEMGUARD, pathname_regex, sym_name, handler_func, original_func); + if (UNLIKELY(ret != 0)) { + LOGE(LOG_TAG, "Fail to hook symbol '%s' of libs match pattern '%s', ret: %d", sym_name, pathname_regex, ret); + return false; + } + LOGD(LOG_TAG, "Success! DoHook(%s, %s, %p, %p)", pathname_regex, sym_name, handler_func, original_func); + return true; +} + +bool memguard::EndHook(const std::vector& ignore_pathname_regex_list) { + int ret = 0; + for (auto & pattern : ignore_pathname_regex_list) { + if ((ret = xhook_grouped_ignore(HOOK_REQUEST_GROUPID_MEMGUARD, pattern.c_str(), nullptr)) != 0) { + LOGE(LOG_TAG, "Fail to ignore all symbols in library matches pattern %s, ret: %d", pattern.c_str(), ret); + return false; + } + } + if ((ret = xhook_grouped_ignore(HOOK_REQUEST_GROUPID_MEMGUARD, ".*/libmemguard\\.so$", nullptr)) != 0) { + LOGE(LOG_TAG, "Fail to ignore all symbols in libmemguard.so, ret: %d", ret); + return false; + } +#if defined(__LP64__) + if ((ret = xhook_grouped_ignore(HOOK_REQUEST_GROUPID_MEMGUARD, ".*/linker64$", nullptr)) != 0) { + LOGE(LOG_TAG, "Fail to ignore all symbols in linker64, ret: %d", ret); + return false; + } +#else + if ((ret = xhook_grouped_ignore(HOOK_REQUEST_GROUPID_MEMGUARD, ".*/linker$", nullptr)) != 0) { + LOGE(LOG_TAG, "Fail to ignore all symbols in linker, ret: %d", ret); + return false; + } +#endif + if ((ret = xhook_grouped_ignore(HOOK_REQUEST_GROUPID_MEMGUARD, ".*/libc\\.so", nullptr)) != 0) { + LOGE(LOG_TAG, "Fail to ignore all symbols in libc.so, ret: %d", ret); + return false; + } + xhook_enable_debug(0); + xhook_enable_sigsegv_protection(1); + if (!UpdateHook()) { + return false; + } + LOGD(LOG_TAG, "EndHook()"); + return true; +} + +bool memguard::UpdateHook() { + int ret = 0; + if ((ret = xhook_refresh(0)) != 0) { + LOGE(LOG_TAG, "Fail to call xhook_refresh, ret: %d", ret); + return false; + } + LOGD(LOG_TAG, "UpdateHook()"); + return true; +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Log.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Log.cpp new file mode 100644 index 000000000..5b872d9dc --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Log.cpp @@ -0,0 +1,21 @@ +// +// Created by tomystang on 2020/11/24. +// + +#include +#include +// #include + +using namespace memguard; + +void memguard::log::PrintLog(int level, const char* tag, const char* fmt, ...) { + va_list args; + va_start(args, fmt); + PrintLogV(level, tag, fmt, args); + va_end(args); +} + +void memguard::log::PrintLogV(int level, const char* tag, const char* fmt, va_list args) { + __android_log_vprint(level, tag, fmt, args); + // internal_hook_vlogger(level, tag, fmt, args); +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Memory.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Memory.cpp new file mode 100644 index 000000000..9de87364c --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Memory.cpp @@ -0,0 +1,72 @@ +// +// Created by tomystang on 2020/11/26. +// + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace memguard; + +#define LOG_TAG "MemGuard.Memory" + +#define PR_SET_VMA 0x53564d41 +#define PR_SET_VMA_ANON_NAME 0 + +void* memguard::memory::MapMemArea(size_t size, const char* region_name, void* hint, bool force_hint) { + int flags = MAP_PRIVATE | MAP_ANONYMOUS; + if (force_hint) { + flags |= MAP_FIXED; + } + #ifdef __NR_mmap2 + void* res = (void*) syscall(__NR_mmap2, hint, size, PROT_NONE, flags, -1, 0); + #else + void* res = (void*) syscall(__NR_mmap, hint, size, gOpts.ignoreOverlappedReading ? PROT_READ : PROT_NONE, flags, -1, 0); + #endif + if (res == (void*) -1) { + return nullptr; + } +#ifdef __ANDROID__ + if (syscall(__NR_prctl, PR_SET_VMA, PR_SET_VMA_ANON_NAME, + (unsigned long) res, (unsigned long) size, (unsigned long) region_name) != 0) { + int errcode = errno; + LOGW(LOG_TAG, "Fail to name anonymous mmaped region, error: %s(%d)", strerror(errcode), errcode); + } +#endif + return res; +} + +void memguard::memory::UnmapMemArea(void* addr, size_t size) { + int res = syscall(__NR_munmap, addr, size); + if (res != 0) { + int errcode = errno; + LOGW(LOG_TAG, "Fail to unmap region, error: %s(%d)", strerror(errcode), errcode); + } +} + +void memguard::memory::MarkAreaReadWrite(void* start, size_t size) { + if (syscall(__NR_mprotect, start, size, PROT_READ | PROT_WRITE) != 0) { + int errcode = errno; + LOGF(LOG_TAG, "Fail to change privilege of mark region (%p ~ +%" PRIu32 ") to 'rw', error: %s(%d)", + start, size, strerror(errcode), errcode); + } +} + +void memguard::memory::MarkAreaInaccessible(void* start, size_t size) { + if (syscall(__NR_mprotect, start, size, gOpts.ignoreOverlappedReading ? PROT_READ : PROT_NONE) != 0) { + int errcode = errno; + LOGF(LOG_TAG, "Fail to change privilege of mark region (%p ~ +%" PRIu32 ") to 'none', error: %s(%d)", + start, size, strerror(errcode), errcode); + } +} + +size_t memguard::memory::GetPageSize() { + return sysconf(_SC_PAGESIZE); +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Mutex.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Mutex.cpp new file mode 100644 index 000000000..9140f818b --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Mutex.cpp @@ -0,0 +1,40 @@ +// +// Created by tomystang on 2020/11/26. +// + +#include +#include +#include +#include + +using namespace memguard; + +#define LOG_TAG "MemGuard.Mutex" + +bool memguard::Mutex::tryLock() { + int ret = pthread_mutex_trylock(&mRawMutex); + if (LIKELY(ret == 0)) { + return true; + } else { + if (ret != EBUSY) { + LOGE(LOG_TAG, "Fail to hold mutex without blocking, error: %s(%d)", strerror(ret), ret); + } + return false; + } +} + +void memguard::Mutex::lock() { + int ret = pthread_mutex_lock(&mRawMutex); + if (ret != 0) { + int errcode = errno; + LOGE(LOG_TAG, "Fail to hold mutex, error: %s(%d)", strerror(errcode), errcode); + } +} + +void memguard::Mutex::unlock() { + int ret = pthread_mutex_unlock(&mRawMutex); + if (ret != 0) { + int errcode = errno; + LOGE(LOG_TAG, "Fail to hold mutex, error: %s(%d)", strerror(errcode), errcode); + } +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Paths.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Paths.cpp new file mode 100644 index 000000000..2b6be39d9 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Paths.cpp @@ -0,0 +1,37 @@ +// +// Created by tomystang on 2021/2/2. +// + +#include +#include +#include +#include +#include + +using namespace memguard; + +bool memguard::paths::Exists(const std::string &path) { + return syscall(__NR_faccessat, AT_FDCWD, path.c_str(), F_OK) == 0; +} + +std::string memguard::paths::GetParent(const std::string& path) { + auto lastSlashPos = path.find_last_of("/\\"); + if (lastSlashPos != std::string::npos) { + return path.substr(0, lastSlashPos); + } else { + return path; + } +} + +bool memguard::paths::MakeDirs(const std::string& path, int mode) { + if (Exists(path)) { + return true; + } + if (!MakeDirs(GetParent(path))) { + return false; + } + if (syscall(__NR_mkdirat, AT_FDCWD, path.c_str(), 0000) != 0) { + return false; + } + return syscall(__NR_fchmodat, AT_FDCWD, path.c_str(), mode, 0) == 0; +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Random.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Random.cpp new file mode 100644 index 000000000..5a976a08d --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Random.cpp @@ -0,0 +1,22 @@ +// +// Created by tomystang on 2020/11/26. +// + +#include +#include +#include + +using namespace memguard; + +static TLSVAR uint32_t sLastRndValue = 0xFCDE97AB; + +void memguard::random::InitializeRndSeed() { + sLastRndValue = ::time(nullptr) + ::gettid(); +} + +uint32_t memguard::random::GenerateUnsignedInt32() { + sLastRndValue ^= sLastRndValue << 7; + sLastRndValue ^= sLastRndValue >> 17; + sLastRndValue ^= sLastRndValue << 5; + return sLastRndValue; +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Unwind.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Unwind.cpp new file mode 100644 index 000000000..350b577cf --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/port/Unwind.cpp @@ -0,0 +1,60 @@ +// +// Created by tomystang on 2020/10/15. +// + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace memguard; +using namespace unwindstack; + +static std::optional sLocalUnwinder; + +bool memguard::unwind::Initialize() { + sLocalUnwinder.emplace(); + return sLocalUnwinder->Init(); +} + +int memguard::unwind::UnwindStack(void** pcs, size_t max_count) { +#if defined(__aarch64__) || defined(__i386__) + return fastunwind::Unwind(pcs, max_count); +#else + return sLocalUnwinder->Unwind(pcs, max_count); +#endif +} + +int memguard::unwind::UnwindStack(void* ucontext, void** pcs, size_t max_count) { +#if defined(__aarch64__) || defined(__i386__) + return fastunwind::Unwind(ucontext, pcs, max_count); +#else + return sLocalUnwinder->Unwind(ucontext, pcs, max_count); +#endif +} + +char* memguard::unwind::GetStackElementDescription(const void* pc, char* desc_out, size_t max_length) { + std::string desc; + size_t outputLen = max_length - 1; + if (sLocalUnwinder->GetStackElementString((uintptr_t) pc, &desc)) { + if (desc.length() < outputLen) { + outputLen = desc.length(); + } + strncpy(desc_out, desc.c_str(), outputLen); + } else { + const char* failureStr = ""; + size_t failureStrLen = strlen(failureStr); + if (failureStrLen < outputLen) { + outputLen = failureStrLen; + } + strncpy(desc_out, failureStr, outputLen); + } + desc_out[outputLen] = '\0'; + return desc_out; +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Allocation.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Allocation.cpp new file mode 100644 index 000000000..4d28a8520 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Allocation.cpp @@ -0,0 +1,125 @@ +// +// Created by tomystang on 2020/10/16. +// + +#include +#include "PagePool.h" +#include "Auxiliary.h" +#include "Allocation.h" +#include "Options.h" + +using namespace memguard; + +static size_t sAdjustedPercentageOfLeftSideGuard = 0; +static size_t sSampleCycleLength = 0; + +struct alignas(8) TLSValues { + uint32_t sampleCounter = 0; + bool isCallingAlloc = false; + bool isCallingFree = false; +}; + +static TLSValues TLSVAR sTLSValues; + +static bool SUPRESS_UNUSED ShouldBeGuardedThisTime() { + if (UNLIKELY(sTLSValues.sampleCounter == 0)) { + sTLSValues.sampleCounter = (random::GenerateUnsignedInt32() & (sSampleCycleLength - 1)); + return true; + } else { + --sTLSValues.sampleCounter; + return false; + } +} + +static bool SUPRESS_UNUSED ShouldBeGuardedOnLeft() { + // rnd % 1024 <= percentage / 100 * 1024 + // ==> rnd % 1024 * 100 <= percentage * 1024 + return (memguard::random::GenerateUnsignedInt32() & ((1U << 10U) - 1)) * 100U + <= (sAdjustedPercentageOfLeftSideGuard << 10U); +} + +#define RECURSIVE_GUARD(tls_var, ret_value_on_reentered) \ + if (tls_var) { \ + return (ret_value_on_reentered); \ + } \ + (tls_var) = true; \ + ON_SCOPE_EXIT((tls_var) = false) + + +bool memguard::allocation::Prepare() { + sAdjustedPercentageOfLeftSideGuard = gOpts.percentageOfLeftSideGuard; + if (sAdjustedPercentageOfLeftSideGuard > 100) { + sAdjustedPercentageOfLeftSideGuard = 100; + } + + sSampleCycleLength = AlignUpToPowOf2(gOpts.maxSkippedAllocationCount); + if (sSampleCycleLength == 0) { + sSampleCycleLength = 1; + } else if (sSampleCycleLength == 1) { + sSampleCycleLength = 2; + } + + random::InitializeRndSeed(); + + sTLSValues.sampleCounter = (random::GenerateUnsignedInt32() & (sSampleCycleLength - 1)); + + return true; +} + +void* memguard::allocation::Allocate(size_t size) { + return AlignedAllocate(size, (sizeof(void*) << 1)); +} + +void* memguard::allocation::AlignedAllocate(size_t size, size_t alignment) { + RECURSIVE_GUARD(sTLSValues.isCallingAlloc, nullptr); + + if (!ShouldBeGuardedThisTime()) { + return nullptr; + } + + if (UNLIKELY(alignment > pagepool::GetSlotSize())) { + return nullptr; + } + + if (UNLIKELY(!IsPowOf2(alignment))) { + return nullptr; + } + + if (UNLIKELY(AlignUpTo(size, alignment) > pagepool::GetSlotSize())) { + return nullptr; + } + + pagepool::GuardSide guardSide = ShouldBeGuardedOnLeft() + ? pagepool::GuardSide::ON_LEFT + : pagepool::GuardSide::ON_RIGHT; + int slot = pagepool::BorrowSlot(size, alignment, guardSide); + if (slot == pagepool::SLOT_NONE) { + return nullptr; + } + + return pagepool::GetAllocatedAddress(slot); +} + +bool memguard::allocation::IsAllocatedByThisAllocator(void* ptr) { + return pagepool::GetSlotIdOfAddress(ptr) != pagepool::SLOT_NONE; +} + +size_t memguard::allocation::GetAllocatedSize(void* ptr) { + int slot = pagepool::GetSlotIdOfAddress(ptr); + if (slot == pagepool::SLOT_NONE) { + return 0; + } + return pagepool::GetAllocatedSize(slot); +} + +bool memguard::allocation::Free(void* ptr) { + RECURSIVE_GUARD(sTLSValues.isCallingFree, false); + + int slot = pagepool::GetSlotIdOfAddress(ptr); + if (slot != pagepool::SLOT_NONE) { + pagepool::ReturnSlot(slot); + return true; + } else { + return false; + } +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Allocation.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Allocation.h new file mode 100644 index 000000000..5335059ca --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Allocation.h @@ -0,0 +1,23 @@ +// +// Created by tomystang on 2020/10/16. +// + +#ifndef __MEMGUARD_MEMALLOCATION_H__ +#define __MEMGUARD_MEMALLOCATION_H__ + + +#include + +namespace memguard { + namespace allocation { + extern bool Prepare(); + extern void* Allocate(size_t size); + extern void* AlignedAllocate(size_t size, size_t alignment); + extern bool IsAllocatedByThisAllocator(void* ptr); + extern size_t GetAllocatedSize(void* ptr); + extern bool Free(void* ptr); + } +} + + +#endif //__MEMGUARD_MEMALLOCATION_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Auxiliary.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Auxiliary.h new file mode 100644 index 000000000..68e44f9ee --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Auxiliary.h @@ -0,0 +1,183 @@ +// +// Created by tomystang on 2020/11/27. +// + +#ifndef __MEMGUARD_AUXILIARY_H__ +#define __MEMGUARD_AUXILIARY_H__ + + +#include + +#ifndef TLSVAR +#define TLSVAR __thread __attribute__((tls_model("initial-exec"))) +#endif + +#ifndef LIKELY +#define LIKELY(cond) __builtin_expect(!!(cond), 1) +#endif + +#ifndef UNLIKELY +#define UNLIKELY(cond) __builtin_expect(!!(cond), 0) +#endif + +#ifndef UNUSED +#define UNUSED(v) ((void) (v)) +#endif + +#ifndef SUPRESS_UNUSED +#define SUPRESS_UNUSED __attribute__((unused)) +#endif + +#define DISALLOW_COPY(type) \ + private: \ + type(const type&) = delete; \ + type& operator =(const type&) = delete + +#define DISALLOW_COPY_ASSIGN(type) \ + private: \ + type& operator =(const type&) = delete + +#define DISALLOW_MOVE(type) \ + private: \ + type(type&&) = delete; \ + type& operator =(type&&) = delete + +#define DISALLOW_MOVE_ASSIGN(type) \ + private: \ + type& operator =(type&&) = delete + +#define DISALLOW_NEW(type) \ + private: \ + void* operator new(size_t) = delete; \ + void* operator new[](size_t) = delete + +#define ON_SCOPE_EXIT(impl) auto ON_SCOPE_EXIT = memguard::MakeScopeCleaner([&]() { impl; }) + + +namespace memguard { + template + class ScopeCleaner { + public: + explicit ScopeCleaner(TDtor&& dtor): mDtor(std::forward(dtor)), mOmitted(false) {} + + ScopeCleaner(ScopeCleaner&& other): mDtor(other.mDtor), mOmitted(other.mOmitted) { + other.mOmitted = true; + } + + ~ScopeCleaner() { + if (!mOmitted) { + mDtor(); + mOmitted = true; + } + } + + void omit() { + mOmitted = true; + } + + bool isOmitted() const { + return mOmitted; + } + + private: + DISALLOW_COPY(ScopeCleaner); + DISALLOW_NEW(ScopeCleaner); + DISALLOW_MOVE_ASSIGN(ScopeCleaner); + + TDtor mDtor; + bool mOmitted; + }; + + template + static ScopeCleaner MakeScopeCleaner(TDtor&& dtor) { + return ScopeCleaner(std::forward(dtor)); + } + + template + class ScopedObject { + public: + ScopedObject(TObj&& obj, TDtor&& dtor) + : mObj(std::forward(obj)), mDtor(std::forward(dtor)), mDetached(false) {} + + ScopedObject(ScopedObject&& other) + : mObj(other.detach()), mDtor(other.mDtor), mDetached(false) {} + + ScopedObject& operator =(ScopedObject&& rhs) { + mObj = std::move(rhs.mObj); + mDtor = std::move(rhs.mDtor); + mDetached = rhs.mDetached; + return *this; + } + + ~ScopedObject() { + if (!mDetached) { + mDtor(mObj); + mDetached = true; + } + } + + TObj& get() { + return mObj; + } + + TObj& detach() { + mDetached = true; + return mObj; + } + + bool isDetached() const { + return mDetached; + } + + private: + DISALLOW_COPY(ScopedObject); + DISALLOW_NEW(ScopedObject); + + TObj mObj; + TDtor mDtor; + bool mDetached; + }; + + template + static ScopedObject MakeScopedObject(TObj&& obj, TDtor&& dtor) { + return ScopedObject(std::forward(obj), std::forward(dtor)); + } + + namespace random { + extern void InitializeRndSeed(); + extern uint32_t GenerateUnsignedInt32(); + } + + static inline bool IsPowOf2(uint64_t value) { + return (value & (value - 1)) == 0; + } + + static inline uint64_t AlignUpTo(uint64_t value, size_t align) { + if (LIKELY(IsPowOf2(align))) { + return ((value + align - 1) & (~(align - 1))); + } else { + return ((value + align - 1) / align * align); + } + } + + static inline uint64_t AlignDownTo(uint64_t value, size_t align) { + if (LIKELY(IsPowOf2(align))) { + return ((value) & (~(align - 1))); + } else { + return (value / align * align); + } + } + + static inline uint64_t AlignUpToPowOf2(uint64_t value) { + --value; + value |= value >> 1; + value |= value >> 2; + value |= value >> 4; + value |= value >> 8; + value |= value >> 16; + return ++value; + } +} + + +#endif //__MEMGUARD_AUXILIARY_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/FdSanWrapper.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/FdSanWrapper.h new file mode 100644 index 000000000..090a40c93 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/FdSanWrapper.h @@ -0,0 +1,41 @@ +// +// Created by YinSheng Tang on 2021/9/22. +// + +#ifndef MATRIX_ANDROID_FDSANWRAPPER_H +#define MATRIX_ANDROID_FDSANWRAPPER_H + + +enum android_fdsan_error_level { + // No errors. + ANDROID_FDSAN_ERROR_LEVEL_DISABLED, + // Warn once(ish) on error, and then downgrade to ANDROID_FDSAN_ERROR_LEVEL_DISABLED. + ANDROID_FDSAN_ERROR_LEVEL_WARN_ONCE, + // Warn always on error. + ANDROID_FDSAN_ERROR_LEVEL_WARN_ALWAYS, + // Abort on error. + ANDROID_FDSAN_ERROR_LEVEL_FATAL, +}; + +/* + * Get the error level. + */ +extern android_fdsan_error_level AndroidFdSanGetErrorLevel(); + +/* + * Set the error level and return the previous state. + * + * Error checking is automatically disabled in the child of a fork, to maintain + * compatibility with code that forks, blindly closes FDs, and then execs. + * + * In cases such as the zygote, where the child has no intention of calling + * exec, call this function to reenable fdsan checks. + * + * This function is not thread-safe and does not synchronize with checks of the + * value, and so should probably only be called in single-threaded contexts + * (e.g. postfork). + */ +extern android_fdsan_error_level AndroidFdSanSetErrorLevel(enum android_fdsan_error_level new_level); + + +#endif //MATRIX_ANDROID_FDSANWRAPPER_H diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Hook.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Hook.h new file mode 100644 index 000000000..01d3179a1 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Hook.h @@ -0,0 +1,20 @@ +// +// Created by tomystang on 2020/10/15. +// + +#ifndef __MEMGUARD_HOOK_H__ +#define __MEMGUARD_HOOK_H__ + + +#include +#include + +namespace memguard { + extern bool BeginHook(); + extern bool DoHook(const char* pathname_regex, const char* sym_name, void* handler_func, void** original_func); + extern bool EndHook(const std::vector& ignore_pathname_regex_list); + extern bool UpdateHook(); +} + + +#endif //__MEMGUARD_HOOK_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Interception.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Interception.cpp new file mode 100644 index 000000000..db67f9f11 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Interception.cpp @@ -0,0 +1,620 @@ +// +// Created by tomystang on 2020/10/16. +// +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "Auxiliary.h" +#include "Allocation.h" +#include "Hook.h" +#include "Interception.h" +#include "Log.h" +#include "Memory.h" + +using namespace memguard; + +#define LOG_TAG "MemGuard.Interception" + +#if defined(__LP64__) +#define DEFAULT_PTR_ALIGN 16 +#else +#define DEFAULT_PTR_ALIGN 8 +#endif + +static void* sLibCHandle = nullptr; +static void* sSelfLibHandle = nullptr; + +static void* HandleMAlloc(size_t size); +static void* HandleCAlloc(size_t elem_count, size_t elem_size); +static void* HandleReAlloc(void* ptr, size_t size); +static void HandleFree(void* ptr); +static void* HandleMemAlign(size_t alignment, size_t byte_count); +static int HandlePosixMemAlign(void** ptr, size_t alignment, size_t size); +static void* HandleAlignedAlloc(size_t alignment, size_t size); +static void* HandleVAlloc(size_t size); +static void* HandlePVAlloc(size_t size); +static char* HandleStrDup(const char* str); +static char* HandleStrNDup(const char* str, size_t n); +static size_t HandleMAllocUsableSize(const void* ptr); +static void HandleFree(void* ptr); +static void* HandleNew(size_t size); +static void* HandleNewNoThrow(size_t size, const std::nothrow_t&); +static void* HandleNewAligned(size_t size, size_t align); +static void* HandleNewAlignedNoThrow(size_t size, size_t align, const std::nothrow_t&); +static void* HandleNewArr(size_t size); +static void* HandleNewArrNoThrow(size_t size, const std::nothrow_t&); +static void* HandleNewArrAligned(size_t size, size_t align); +static void* HandleNewArrAlignedNoThrow(size_t size, size_t align, const std::nothrow_t&); +static void HandleDelete(void* ptr); +static void HandleDeleteNoThrow(void* ptr, const std::nothrow_t&); +static void HandleDeleteAligned(void* ptr, size_t align); +static void HandleDeleteAlignedNoThrow(void* ptr, size_t align, const std::nothrow_t&); +static void HandleDeleteSize(void* ptr, unsigned int size); +static void HandleDeleteSizeAligned(void* ptr, unsigned int size, size_t align); +static void HandleDeleteArr(void* ptr); +static void HandleDeleteArrNoThrow(void* ptr, const std::nothrow_t&); +static void HandleDeleteArrAligned(void* ptr, size_t align); +static void HandleDeleteArrAlignedNoThrow(void* ptr, size_t align, const std::nothrow_t&); +static void HandleDeleteArrSize(void* ptr, unsigned int size); +static void HandleDeleteArrSizeAligned(void* ptr, unsigned int size, size_t align); + +#ifdef __LP64__ + #define SYM_NEW _Znwm + #define SYM_NEW_NOTHROW _ZnwmRKSt9nothrow_t + #define SYM_NEW_ALIGNED _ZnwmSt11align_val_t + #define SYM_NEW_ALIGNED_NOTHROW _ZnwmSt11align_val_tRKSt9nothrow_t + #define SYM_NEW_ARR _Znam + #define SYM_NEW_ARR_NOTHROW _ZnamRKSt9nothrow_t + #define SYM_NEW_ARR_ALIGNED _ZnamSt11align_val_t + #define SYM_NEW_ARR_ALIGNED_NOTHROW _ZnamSt11align_val_tRKSt9nothrow_t + #define SYM_DELETE_SIZE _ZdlPvm + #define SYM_DELETE_SIZE_ALIGNED _ZdlPvmSt11align_val_t + #define SYM_DELETE_ARR_SIZE _ZdaPvm + #define SYM_DELETE_ARR_SIZE_ALIGNED _ZdaPvmSt11align_val_t +#else + #define SYM_NEW _Znwj + #define SYM_NEW_NOTHROW _ZnwjRKSt9nothrow_t + #define SYM_NEW_ALIGNED _ZnwjSt11align_val_t + #define SYM_NEW_ALIGNED_NOTHROW _ZnwjSt11align_val_tRKSt9nothrow_t + #define SYM_NEW_ARR _Znaj + #define SYM_NEW_ARR_NOTHROW _ZnajRKSt9nothrow_t + #define SYM_NEW_ARR_ALIGNED _ZnajSt11align_val_t + #define SYM_NEW_ARR_ALIGNED_NOTHROW _ZnajSt11align_val_tRKSt9nothrow_t + #define SYM_DELETE_SIZE _ZdlPvj + #define SYM_DELETE_SIZE_ALIGNED _ZdlPvjSt11align_val_t + #define SYM_DELETE_ARR_SIZE _ZdaPvj + #define SYM_DELETE_ARR_SIZE_ALIGNED _ZdaPvjSt11align_val_t +#endif + +#define SYM_DELETE _ZdlPv +#define SYM_DELETE_NOTHROW _ZdlPvRKSt9nothrow_t +#define SYM_DELETE_ALIGNED _ZdlPvSt11align_val_t +#define SYM_DELETE_ALIGNED_NOTHROW _ZdlPvSt11align_val_tRKSt9nothrow_t +#define SYM_DELETE_ARR _ZdaPv +#define SYM_DELETE_ARR_NOTHROW _ZdaPvRKSt9nothrow_t +#define SYM_DELETE_ARR_ALIGNED _ZdaPvSt11align_val_t +#define SYM_DELETE_ARR_ALIGNED_NOTHROW _ZdaPvSt11align_val_tRKSt9nothrow_t + +#define _X_6(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) \ + _X_5(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) +#define _X_5(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) \ + _X_4(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) +#define _X_4(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) \ + _X_3(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) +#define _X_3(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) \ + _X_2(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) +#define _X_2(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) \ + _X_1(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) +#define _X_1(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) \ + X(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) + +#define ENUM_C_ALLOC_FUNCTIONS(target_lib_pattern) \ + _X_6(target_lib_pattern, void*, malloc, (size_t), HandleMAlloc, sLibCHandle) \ + _X_6(target_lib_pattern, void*, calloc, (size_t, size_t), HandleCAlloc, sLibCHandle) \ + _X_6(target_lib_pattern, void*, realloc, (void*, size_t), HandleReAlloc, sLibCHandle) \ + _X_6(target_lib_pattern, void*, memalign, (size_t, size_t), HandleMemAlign, sLibCHandle) \ + _X_6(target_lib_pattern, int, posix_memalign, (void**, size_t, size_t), HandlePosixMemAlign, sLibCHandle) \ + _X_6(target_lib_pattern, void*, aligned_alloc, (size_t, size_t), HandleAlignedAlloc, sLibCHandle) \ + _X_6(target_lib_pattern, char*, strdup, (const char*), HandleStrDup, sLibCHandle) \ + _X_6(target_lib_pattern, char*, strndup, (const char*, size_t), HandleStrNDup, sLibCHandle) + +#define ENUM_C_DEALLOC_AND_OTHER_FUNCTIONS(target_lib_pattern) \ + _X_6(target_lib_pattern, size_t, malloc_usable_size, (const void*), HandleMAllocUsableSize, sLibCHandle) \ + _X_6(target_lib_pattern, void, free, (void*), HandleFree, sLibCHandle) + +#define ENUM_CPP_ALLOC_FUNCTIONS(target_lib_pattern) \ + _X_6(target_lib_pattern, void*, SYM_NEW, (size_t), HandleNew, sSelfLibHandle) \ + _X_6(target_lib_pattern, void*, SYM_NEW_NOTHROW, (size_t, const std::nothrow_t&), HandleNewNoThrow, sSelfLibHandle) \ + _X_6(target_lib_pattern, void*, SYM_NEW_ALIGNED, (size_t, size_t), HandleNewAligned, sSelfLibHandle) \ + _X_6(target_lib_pattern, void*, SYM_NEW_ALIGNED_NOTHROW, (size_t, size_t, const std::nothrow_t&), HandleNewAlignedNoThrow, sSelfLibHandle) \ + _X_6(target_lib_pattern, void*, SYM_NEW_ARR, (size_t), HandleNewArr, sSelfLibHandle) \ + _X_6(target_lib_pattern, void*, SYM_NEW_ARR_NOTHROW, (size_t, const std::nothrow_t&), HandleNewArrNoThrow, sSelfLibHandle) \ + _X_6(target_lib_pattern, void*, SYM_NEW_ARR_ALIGNED, (size_t, size_t), HandleNewArrAligned, sSelfLibHandle) \ + _X_6(target_lib_pattern, void*, SYM_NEW_ARR_ALIGNED_NOTHROW, (size_t, size_t, const std::nothrow_t&), HandleNewArrAlignedNoThrow, sSelfLibHandle) + +#define ENUM_CPP_DEALLOC_FUNCTIONS(target_lib_pattern) \ + _X_6(target_lib_pattern, void, SYM_DELETE, (void*), HandleDelete, sSelfLibHandle) \ + _X_6(target_lib_pattern, void, SYM_DELETE_NOTHROW, (void*, const std::nothrow_t&), HandleDeleteNoThrow, sSelfLibHandle) \ + _X_6(target_lib_pattern, void, SYM_DELETE_ALIGNED, (void*, size_t), HandleDeleteAligned, sSelfLibHandle) \ + _X_6(target_lib_pattern, void, SYM_DELETE_ALIGNED_NOTHROW, (void*, size_t, const std::nothrow_t&), HandleDeleteAlignedNoThrow, sSelfLibHandle) \ + _X_6(target_lib_pattern, void, SYM_DELETE_ARR, (void*), HandleDeleteArr, sSelfLibHandle) \ + _X_6(target_lib_pattern, void, SYM_DELETE_ARR_NOTHROW, (void*, const std::nothrow_t&), HandleDeleteArrNoThrow, sSelfLibHandle) \ + _X_6(target_lib_pattern, void, SYM_DELETE_ARR_ALIGNED, (void*, size_t), HandleDeleteArrAligned, sSelfLibHandle) \ + _X_6(target_lib_pattern, void, SYM_DELETE_ARR_ALIGNED_NOTHROW, (void*, size_t, const std::nothrow_t&), HandleDeleteArrAlignedNoThrow, sSelfLibHandle) \ + _X_6(target_lib_pattern, void, SYM_DELETE_SIZE, (void*, unsigned int), HandleDeleteSize, sSelfLibHandle) \ + _X_6(target_lib_pattern, void, SYM_DELETE_SIZE_ALIGNED, (void*, unsigned int, size_t), HandleDeleteSizeAligned, sSelfLibHandle) \ + _X_6(target_lib_pattern, void, SYM_DELETE_ARR_SIZE, (void*, unsigned int), HandleDeleteArrSize, sSelfLibHandle) \ + _X_6(target_lib_pattern, void, SYM_DELETE_ARR_SIZE_ALIGNED, (void*, unsigned int, size_t), HandleDeleteArrSizeAligned, sSelfLibHandle) + +static struct { + #define X(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) \ + ret_type (*orig_##sym) args = nullptr; + + ENUM_C_ALLOC_FUNCTIONS(_) + ENUM_C_DEALLOC_AND_OTHER_FUNCTIONS(_) + ENUM_CPP_ALLOC_FUNCTIONS(_) + ENUM_CPP_DEALLOC_FUNCTIONS(_) + + #undef X +} sOriginalFunctions; + +#define ORIGINAL_FUNCTION(sym) _ORIGINAL_FUNCTION_1(sym) +#define _ORIGINAL_FUNCTION_1(sym) sOriginalFunctions.orig_##sym + +static void* HandleMAlloc(size_t size) { + void* res = allocation::Allocate(size); + if (res == nullptr) { + res = ORIGINAL_FUNCTION(malloc)(size); + LOGD(LOG_TAG, "HandleMAlloc(%" PRIu32 ") = %p, skipped.", size, res); + } else { + LOGD(LOG_TAG, "HandleMAlloc(%" PRIu32 ") = %p, guarded.", size, res); + } + return res; +} + +static void* HandleCAlloc(size_t elem_count, size_t elem_size) { + void* res = allocation::Allocate(elem_count * elem_size); + if (res == nullptr) { + res = ORIGINAL_FUNCTION(calloc)(elem_count, elem_size); + LOGD(LOG_TAG, "HandleCAlloc(%" PRIu32 ",%" PRIu32 ") = %p, skipped.", elem_count, elem_size, res); + } else { + memset(res, 0, elem_count * elem_size); + LOGD(LOG_TAG, "HandleCAlloc(%" PRIu32 ",%" PRIu32 ") = %p, guarded.", elem_count, elem_size, res); + } + return res; +} + +static void* HandleReAlloc(void* ptr, size_t size) { + if (ptr == nullptr) { + return HandleMAlloc(size); + } + if (size == 0) { + HandleFree(ptr); + return ptr; + } + if (LIKELY(!allocation::IsAllocatedByThisAllocator(ptr))) { + void* res = ORIGINAL_FUNCTION(realloc)(ptr, size); + LOGD(LOG_TAG, "HandleReAlloc(%p,%" PRIu32 ") = %p, skipped.", ptr, size, res); + return res; + } + + void* newPtr = HandleMAlloc(size); + if (newPtr == nullptr) { + LOGD(LOG_TAG, "HandleReAlloc(%p,%" PRIu32 ") = %p, failure.", ptr, size, newPtr); + errno = ENOMEM; + return nullptr; + } + size_t oldSize = allocation::GetAllocatedSize(ptr); + memcpy(newPtr, ptr, (oldSize <= size ? oldSize : size)); + allocation::Free(ptr); + LOGD(LOG_TAG, "HandleReAlloc(%p,%" PRIu32 ") = %p, moved, guarded.", ptr, size, newPtr); + return newPtr; +} + +static void* HandleMemAlign(size_t alignment, size_t byte_count) { + if (UNLIKELY(!IsPowOf2(alignment))) { + errno = EINVAL; + return nullptr; + } + void* res = allocation::AlignedAllocate(byte_count, alignment); + if (res != nullptr) { + LOGD(LOG_TAG, "HandleMemAlign(%" PRIu32 ",%" PRIu32 ") = %p, guarded.", alignment, byte_count, res); + return res; + } else { + res = ORIGINAL_FUNCTION(memalign)(alignment, byte_count); + LOGD(LOG_TAG, "HandleMemAlign(%" PRIu32 ",%" PRIu32 ") = %p, skipped.", alignment, byte_count, res); + return res; + } +} + +static int HandlePosixMemAlign(void** ptr, size_t alignment, size_t size) { + if (UNLIKELY((alignment & (sizeof(void*) - 1)) != 0)) { + return EINVAL; + } + void* res = allocation::AlignedAllocate(size, alignment); + if (res != nullptr) { + LOGD(LOG_TAG, "HandlePosixMemAlign(%p,%" PRIu32 ",%" PRIu32 ") = 0 (ptr_res:%p), guarded.", + ptr, alignment, size, res); + *ptr = res; + return 0; + } else { + int originalFnRet = ORIGINAL_FUNCTION(posix_memalign)(ptr, alignment, size); + LOGD(LOG_TAG, "HandlePosixMemAlign(%p,%" PRIu32 ",%" PRIu32 ") = %d (ptr_res:%p), skipped.", + ptr, alignment, size, originalFnRet, *ptr); + return originalFnRet; + } +} + +static void* HandleAlignedAlloc(size_t alignment, size_t size) { + if (UNLIKELY(size % alignment != 0)) { + errno = EINVAL; + return nullptr; + } + return HandleMemAlign(alignment, size); +} + +static char* HandleStrDup(const char* str) { + size_t len = strlen(str); + char* buf = (char*) allocation::Allocate(len + 1); + if (buf != nullptr) { + ::memcpy(buf, str, len + 1); + LOGD(LOG_TAG, "HandleStrDup(%p) = %p, guarded.", str, buf); + } else { + buf = ORIGINAL_FUNCTION(strdup)(str); + LOGD(LOG_TAG, "HandleStrDup(%p) = %p, skipped.", str, buf); + } + return buf; +} + +static char* HandleStrNDup(const char* str, size_t n) { + size_t len = strlen(str); + size_t dupLen = (n <= len ? n : len); + char* buf = (char*) allocation::Allocate(dupLen + 1); + if (buf != nullptr) { + ::memcpy(buf, str, dupLen); + buf[dupLen] = '\0'; + LOGD(LOG_TAG, "HandleStrNDup(%p, %" PRIu32 ") = %p, guarded.", str, n, buf); + } else { + buf = ORIGINAL_FUNCTION(strndup)(str, n); + LOGD(LOG_TAG, "HandleStrNDup(%p, %" PRIu32 ") = %p, skipped.", str, n, buf); + } + return buf; +} + +static size_t HandleMAllocUsableSize(const void* ptr) { + size_t res = 0; + if (allocation::IsAllocatedByThisAllocator((void*) ptr)) { + res = allocation::GetAllocatedSize((void*) ptr); + LOGD(LOG_TAG, "HandleMAllocUsableSize(%p) = %" PRIu32 ", guarded.", ptr, res); + } else { + res = ORIGINAL_FUNCTION(malloc_usable_size)(ptr); + LOGD(LOG_TAG, "HandleMAllocUsableSize(%p) = %" PRIu32 ", skipped.", ptr, res); + } + return res; +} + +static void HandleFree(void* ptr) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(free)(ptr); + LOGD(LOG_TAG, "HandleFree(%p), skipped.", ptr); + } else { + LOGD(LOG_TAG, "HandleFree(%p), recycled.", ptr); + } +} + +static void* HandleNew(size_t size) { + void* result = allocation::AlignedAllocate(size, DEFAULT_PTR_ALIGN); + if (result != nullptr) { + LOGD(LOG_TAG, "HandleNew(%" PRIu32 ") = %p, guarded.", size, result); + return result; + } else { + LOGD(LOG_TAG, "HandleNew(%" PRIu32 ") = %p, skipped.", size, result); + return ORIGINAL_FUNCTION(SYM_NEW)(size); + } +} + +static void* HandleNewNoThrow(size_t size, const std::nothrow_t&) { + void* result = allocation::AlignedAllocate(size, DEFAULT_PTR_ALIGN); + if (result != nullptr) { + LOGD(LOG_TAG, "HandleNewNoThrow(%" PRIu32 ") = %p, guarded.", size, result); + return result; + } else { + LOGD(LOG_TAG, "HandleNewNoThrow(%" PRIu32 ") = %p, skipped.", size, result); + return ORIGINAL_FUNCTION(SYM_NEW_NOTHROW)(size, std::nothrow); + } +} + +static void* HandleNewAligned(size_t size, size_t align) { + void* result = allocation::AlignedAllocate(size, align); + if (result != nullptr) { + LOGD(LOG_TAG, "HandleNewAligned(%" PRIu32 ", %" PRIu32 ") = %p, guarded.", size, align, result); + return result; + } else { + LOGD(LOG_TAG, "HandleNewAligned(%" PRIu32 ", %" PRIu32 ") = %p, skipped.", size, align, result); + return ORIGINAL_FUNCTION(SYM_NEW_ALIGNED)(size, align); + } +} + +static void* HandleNewAlignedNoThrow(size_t size, size_t align, const std::nothrow_t&) { + void* result = allocation::AlignedAllocate(size, align); + if (result != nullptr) { + LOGD(LOG_TAG, "HandleNewAlignedNoThrow(%" PRIu32 ", %" PRIu32 ") = %p, guarded.", size, align, result); + return result; + } else { + LOGD(LOG_TAG, "HandleNewAlignedNoThrow(%" PRIu32 ", %" PRIu32 ") = %p, skipped.", size, align, result); + return ORIGINAL_FUNCTION(SYM_NEW_ALIGNED_NOTHROW)(size, align, std::nothrow); + } +} + +static void* HandleNewArr(size_t size) { + void* result = allocation::AlignedAllocate(size, DEFAULT_PTR_ALIGN); + if (result != nullptr) { + LOGD(LOG_TAG, "HandleNewArr(%" PRIu32 ") = %p, guarded.", size, result); + return result; + } else { + LOGD(LOG_TAG, "HandleNewArr(%" PRIu32 ") = %p, skipped.", size, result); + return ORIGINAL_FUNCTION(SYM_NEW_ARR)(size); + } +} + +static void* HandleNewArrNoThrow(size_t size, const std::nothrow_t&) { + void* result = allocation::AlignedAllocate(size, DEFAULT_PTR_ALIGN); + if (result != nullptr) { + LOGD(LOG_TAG, "HandleNewArrNoThrow(%" PRIu32 ") = %p, guarded.", size, result); + return result; + } else { + LOGD(LOG_TAG, "HandleNewArrNoThrow(%" PRIu32 ") = %p, skipped.", size, result); + return ORIGINAL_FUNCTION(SYM_NEW_ARR_NOTHROW)(size, std::nothrow); + } +} + +static void* HandleNewArrAligned(size_t size, size_t align) { + void* result = allocation::AlignedAllocate(size, align); + if (result != nullptr) { + LOGD(LOG_TAG, "HandleNewArrAligned(%" PRIu32 ", %" PRIu32 ") = %p, guarded.", size, align, result); + return result; + } else { + LOGD(LOG_TAG, "HandleNewArrAligned(%" PRIu32 ", %" PRIu32 ") = %p, guarded.", size, align, result); + return ORIGINAL_FUNCTION(SYM_NEW_ARR_ALIGNED)(size, align); + } +} + +static void* HandleNewArrAlignedNoThrow(size_t size, size_t align, const std::nothrow_t&) { + void* result = allocation::AlignedAllocate(size, align); + if (result != nullptr) { + LOGD(LOG_TAG, "HandleNewArrAlignedNoThrow(%" PRIu32 ", %" PRIu32 ") = %p, guarded.", size, align, result); + return result; + } else { + LOGD(LOG_TAG, "HandleNewArrAlignedNoThrow(%" PRIu32 ", %" PRIu32 ") = %p, guarded.", size, align, result); + return ORIGINAL_FUNCTION(SYM_NEW_ARR_ALIGNED_NOTHROW)(size, align, std::nothrow); + } +} + +static void HandleDelete(void* ptr) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(SYM_DELETE)(ptr); + LOGD(LOG_TAG, "HandleDelete(%p), skipped.", ptr); + } else { + LOGD(LOG_TAG, "HandleDelete(%p), recycled.", ptr); + } +} + +static void HandleDeleteNoThrow(void* ptr, const std::nothrow_t&) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(SYM_DELETE_NOTHROW)(ptr, std::nothrow); + LOGD(LOG_TAG, "HandleDeleteNoThrow(%p), skipped.", ptr); + } else { + LOGD(LOG_TAG, "HandleDeleteNoThrow(%p), recycled.", ptr); + } +} + +static void HandleDeleteAligned(void* ptr, size_t align) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(SYM_DELETE_ALIGNED)(ptr, align); + LOGD(LOG_TAG, "HandleDeleteAligned(%p, %" PRIu32 "), skipped.", ptr, align); + } else { + LOGD(LOG_TAG, "HandleDeleteAligned(%p, %" PRIu32 "), recycled.", ptr, align); + } +} + +static void HandleDeleteAlignedNoThrow(void* ptr, size_t align, const std::nothrow_t&) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(SYM_DELETE_ALIGNED_NOTHROW)(ptr, align, std::nothrow); + LOGD(LOG_TAG, "HandleDeleteAlignedNoThrow(%p, %" PRIu32 "), skipped.", ptr, align); + } else { + LOGD(LOG_TAG, "HandleDeleteAlignedNoThrow(%p, %" PRIu32 "), recycled.", ptr, align); + } +} + +static void HandleDeleteSize(void* ptr, unsigned int size) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(SYM_DELETE_SIZE)(ptr, size); + LOGD(LOG_TAG, "HandleDeleteSize(%p, %" PRIu32 "), skipped.", ptr, size); + } else { + LOGD(LOG_TAG, "HandleDeleteSize(%p, %" PRIu32 "), recycled.", ptr, size); + } +} + +static void HandleDeleteSizeAligned(void* ptr, unsigned int size, size_t align) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(SYM_DELETE_SIZE_ALIGNED)(ptr, size, align); + LOGD(LOG_TAG, "HandleDeleteSizeAligned(%p, %" PRIu32 ", %" PRIu32 "), skipped.", ptr, size, align); + } else { + LOGD(LOG_TAG, "HandleDeleteSizeAligned(%p, %" PRIu32 ", %" PRIu32 "), recycled.", ptr, size, align); + } +} + +static void HandleDeleteArr(void* ptr) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(SYM_DELETE_ARR)(ptr); + LOGD(LOG_TAG, "HandleDeleteArr(%p), skipped.", ptr); + } else { + LOGD(LOG_TAG, "HandleDeleteArr(%p), recycled.", ptr); + } +} + +static void HandleDeleteArrNoThrow(void* ptr, const std::nothrow_t&) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(SYM_DELETE_ARR_NOTHROW)(ptr, std::nothrow); + LOGD(LOG_TAG, "HandleDeleteArrNoThrow(%p), skipped.", ptr); + } else { + LOGD(LOG_TAG, "HandleDeleteArrNoThrow(%p), recycled.", ptr); + } +} + +static void HandleDeleteArrAligned(void* ptr, size_t align) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(SYM_DELETE_ARR_ALIGNED)(ptr, align); + LOGD(LOG_TAG, "HandleDeleteArrAligned(%p, %" PRIu32 "), skipped.", ptr, align); + } else { + LOGD(LOG_TAG, "HandleDeleteArrAligned(%p, %" PRIu32 "), recycled.", ptr, align); + } +} + +static void HandleDeleteArrAlignedNoThrow(void* ptr, size_t align, const std::nothrow_t&) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(SYM_DELETE_ARR_ALIGNED_NOTHROW)(ptr, align, std::nothrow); + LOGD(LOG_TAG, "HandleDeleteArrAlignedNoThrow(%p, %" PRIu32 "), skipped.", ptr, align); + } else { + LOGD(LOG_TAG, "HandleDeleteArrAlignedNoThrow(%p, %" PRIu32 "), recycled.", ptr, align); + } +} + +static void HandleDeleteArrSize(void* ptr, unsigned int size) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(SYM_DELETE_ARR_SIZE)(ptr, size); + LOGD(LOG_TAG, "HandleDeleteArrSize(%p, %" PRIu32 "), skipped.", ptr, size); + } else { + LOGD(LOG_TAG, "HandleDeleteArrSize(%p, %" PRIu32 "), recycled.", ptr, size); + } +} + +static void HandleDeleteArrSizeAligned(void* ptr, unsigned int size, size_t align) { + if (!allocation::Free(ptr)) { + ORIGINAL_FUNCTION(SYM_DELETE_ARR_SIZE_ALIGNED)(ptr, size, align); + LOGD(LOG_TAG, "HandleDeleteArrSizeAligned(%p, %" PRIu32 ", %" PRIu32 "), skipped.", ptr, size, align); + } else { + LOGD(LOG_TAG, "HandleDeleteArrSizeAligned(%p, %" PRIu32 ", %" PRIu32 "), recycled.", ptr, size, align); + } +} + +static bool InitializeOriginalFunctions() { + sLibCHandle = ::dlopen("libc.so", RTLD_NOW); + if (sLibCHandle == nullptr) { + LOGE(LOG_TAG, "Fail to get handle of libc.so"); + return false; + } + + sSelfLibHandle = ::dlopen(nullptr, RTLD_NOW); + if (sSelfLibHandle == nullptr) { + LOGE(LOG_TAG, "Fail to get handle of myself."); + return false; + } + + #define X(target_lib_pattern, ret_type, sym, args, handler, def_lib_handle) \ + do { \ + void* hLib = (def_lib_handle); \ + ORIGINAL_FUNCTION(sym) = (ret_type (*) args) dlsym(hLib, #sym); \ + if (UNLIKELY(ORIGINAL_FUNCTION(sym) == nullptr)) { \ + LOGE(LOG_TAG, "Fail to get address of symbol: %s.", #sym); \ + return false; \ + } \ + } while (false); + + ENUM_C_ALLOC_FUNCTIONS(_) + ENUM_C_DEALLOC_AND_OTHER_FUNCTIONS(_) + ENUM_CPP_ALLOC_FUNCTIONS(_) + ENUM_CPP_DEALLOC_FUNCTIONS(_) + + #undef X + + return true; +} + +// static void* (*sOriginalLoaderDlOpen)(const char*, int, const void*) = nullptr; +// static void* HandleLoaderDlOpen(const char* path, int flag, const void* caller_addr) { +// void* result = sOriginalLoaderDlOpen(path, flag, caller_addr); +// if (!UpdateHook()) { +// LOGE(LOG_TAG, "Fail to update hook when load '%s' with flag '%d'", path, flag); +// } +// return result; +// } +// +// static void* (*sOriginalAndroidDlOpenExt)(const char*, int, const void*, const void*) = nullptr; +// static void* HandleAndroidDlOpenExt(const char* path, int flag, const void* extinfo, const void* caller_addr) { +// void* result = sOriginalAndroidDlOpenExt(path, flag, extinfo, caller_addr); +// if (!UpdateHook()) { +// LOGE(LOG_TAG, "Fail to update hook when load '%s' with flag '%d'", path, flag); +// } +// return result; +// } + +#define INTERCEPT_DLOPEN(pattern, failure_ret_val) \ + do { \ + if (!DoHook(pattern, "__loader_dlopen", (void *) HandleLoaderDlOpen, (void **) &sOriginalLoaderDlOpen)) { \ + return (failure_ret_val); \ + } \ + if (!DoHook(pattern, "android_dlopen_ext", (void *) HandleAndroidDlOpenExt, (void **) &sOriginalAndroidDlOpenExt)) { \ + return (failure_ret_val); \ + } \ + } while (false) + +bool memguard::interception::Install() { + if (!InitializeOriginalFunctions()) { + return false; + } + + matrix::PauseLoadSo(); + xhook_block_refresh(); + + auto resume = MakeScopeCleaner([]() { + xhook_unblock_refresh(); + matrix::ResumeLoadSo(); + }); + + if (!BeginHook()) { + return false; + } + + #define X(pattern, ret_type, sym, args, handler, def_lib_handle) \ + if (!DoHook(pattern, #sym, (void*) handler, nullptr)) { \ + return false; \ + } + + for (auto & pattern : gOpts.targetSOPatterns) { + ENUM_C_ALLOC_FUNCTIONS(pattern.c_str()) + ENUM_C_DEALLOC_AND_OTHER_FUNCTIONS(pattern.c_str()) + ENUM_CPP_ALLOC_FUNCTIONS(pattern.c_str()) + ENUM_CPP_DEALLOC_FUNCTIONS(pattern.c_str()) + } + + ENUM_C_DEALLOC_AND_OTHER_FUNCTIONS(".*/libstlport_shared\\.so$") + ENUM_C_DEALLOC_AND_OTHER_FUNCTIONS(".*/libc++_shared\\.so$") + ENUM_C_DEALLOC_AND_OTHER_FUNCTIONS(".*/libc++\\.so$") + ENUM_C_DEALLOC_AND_OTHER_FUNCTIONS(".*/libgnustl_shared\\.so$") + #undef X + + if (xhook_export_symtable_hook("libc.so", "free", reinterpret_cast(HandleFree), nullptr) != 0) { + LOGE(LOG_TAG, "Fail to do export symtab hook for 'free'."); + return false; + } + + // Note: Matrix SoLoadMonitor will handle these dlopen stuffs. + // INTERCEPT_DLOPEN(".*/libnativeloader\\.so$", false); + // INTERCEPT_DLOPEN(".*/libnativeloader_lazy\\.so$", false); + // INTERCEPT_DLOPEN(".*/libopenjdk\\.so$", false); + // INTERCEPT_DLOPEN(".*/libopenjdkjvm\\.so$", false); + // INTERCEPT_DLOPEN(".*/libart\\.so$", false); + // INTERCEPT_DLOPEN(".*/libjavacore\\.so$", false); + // INTERCEPT_DLOPEN(".*/libnativehelper\\.so$", false); + + NOTIFY_COMMON_IGNORE_LIBS(HOOK_REQUEST_GROUPID_MEMGUARD); + + return EndHook(gOpts.ignoredSOPatterns); +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Interception.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Interception.h new file mode 100644 index 000000000..53229b2fe --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Interception.h @@ -0,0 +1,16 @@ +// +// Created by tomystang on 2020/10/16. +// + +#ifndef __MEMGUARD_INTERCEPTION_H__ +#define __MEMGUARD_INTERCEPTION_H__ + + +namespace memguard { + namespace interception { + extern bool Install(); + } +} + + +#endif //__MEMGUARD_INTERCEPTION_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Issue.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Issue.cpp new file mode 100644 index 000000000..69ecac57c --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Issue.cpp @@ -0,0 +1,264 @@ +// +// Created by tomystang on 2020/11/24. +// + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "Auxiliary.h" +#include "Issue.h" +#include "Log.h" +#include "PagePool.h" +#include "Unwind.h" +#include "Memory.h" +#include "Paths.h" + +using namespace memguard; + +#define LOG_TAG "MemGuard.Issue" + +IssueType TLSVAR issue::gLastIssueType = IssueType::UNKNOWN; + +void memguard::issue::TriggerIssue(pagepool::slot_t accessing_slot, IssueType type) { + if (UNLIKELY(accessing_slot == pagepool::SLOT_NONE)) { + return; + } + gLastIssueType = type; + void* rightSideGuardPageAddr = + (void*) ((uintptr_t) pagepool::GetGuardedPageStart(accessing_slot) + pagepool::GetSlotSize()); + // Trigger ACCERR here. + ((char*) rightSideGuardPageAddr)[0] = '\0'; +} + +IssueType memguard::issue::GetLastIssueType() { + return gLastIssueType; +} + +static std::vector GetReadableStackTrace(const void** pcs, size_t count) { + std::vector result; + result.reserve(count); + for (size_t i = 0; i < count; ++i) { + char stack[1024] = {0}; + result.emplace_back(unwind::GetStackElementDescription(pcs[i], stack, 512)); + } + return result; +} + +static void PrintLineV(int fd, const char* fmt, va_list args) { + char line[1024] = {}; + int bytesPrint = vsnprintf(line, sizeof(line), fmt, args); + if (fd >= 0) { + TEMP_FAILURE_RETRY(syscall(__NR_write, fd, line, bytesPrint)); + TEMP_FAILURE_RETRY(syscall(__NR_write, fd, "\n", 1)); + } +} + +static void PrintLine(int fd, const char* fmt, ...) { + va_list args; + va_start(args, fmt); + char line[1024] = {}; + int bytesPrint = vsnprintf(line, sizeof(line), fmt, args); + if (fd >= 0) { + TEMP_FAILURE_RETRY(syscall(__NR_write, fd, line, bytesPrint)); + TEMP_FAILURE_RETRY(syscall(__NR_write, fd, "\n", 1)); + } + va_end(args); +} + +static void PrintStackTrace(int fd, const std::vector& stack_trace, const char* header_fmt, ...) { + va_list headerArgs; + va_start(headerArgs, header_fmt); + ON_SCOPE_EXIT(va_end(headerArgs)); + + PrintLineV(fd, header_fmt, headerArgs); + for (size_t i = 0; i < stack_trace.size(); ++i) { + PrintLine(fd, " #%02d %s", i, stack_trace[i].c_str()); + } + fsync(fd); +} + +static pagepool::slot_t GetNearestSlotID(const void* addr) { + if (addr < pagepool::GetGuardedPageStart(0)) { + return 0; + } + pagepool::slot_t slot = pagepool::GetSlotIdOfAddress(addr); + if (slot != pagepool::SLOT_NONE) { + if (!pagepool::IsAddressInGuardPage(addr)) { + return slot; + } + size_t pageSize = memory::GetPageSize(); + if ((((uintptr_t) addr % pageSize) * 2 > pageSize) + && slot < (pagepool::slot_t) (gOpts.maxDetectableAllocationCount - 1)) { + ++slot; + } + } + return slot; +} + +static char* GetCurrentProcessName(char* buf, size_t buf_size) { + int fd = TEMP_FAILURE_RETRY(syscall(__NR_openat, AT_FDCWD, "/proc/self/cmdline", O_RDONLY, 0)); + if (fd < 0) { + snprintf(buf, buf_size, "?"); + return buf; + } + ON_SCOPE_EXIT(syscall(__NR_close, fd)); + + int bytesRead = TEMP_FAILURE_RETRY(syscall(__NR_read, fd, buf, buf_size - 1)); + buf[bytesRead] = '\0'; + return buf; +} + +static char* GetReadableTimeStamp(char* buf, size_t buf_size) { + timeval tv = {}; + gettimeofday(&tv, nullptr); + uint32_t ms = lrint((float) tv.tv_usec / 1000.0f); + tv.tv_sec += ms / 1000; + ms %= 1000; + + tm tmFields = {}; + localtime_r(&tv.tv_sec, &tmFields); + + size_t writtenLen = strftime(buf, buf_size - 5, "%Y-%m-%d %H:%M:%S", &tmFields); + if (writtenLen < buf_size) { + snprintf(buf + writtenLen, buf_size - writtenLen, ".%03d", ms); + } + + return buf; +} + +bool memguard::issue::Report(const void* accessing_addr, void* ucontext) { + pagepool::slot_t slot = GetNearestSlotID(accessing_addr); + if (slot == pagepool::SLOT_NONE) { + return false; + } + IssueType lastIssueType = GetLastIssueType(); + void* allocatedStart = pagepool::GetAllocatedAddress(slot); + if (lastIssueType == IssueType::UNKNOWN) { + if (pagepool::IsSlotBorrowed(slot)) { + if (allocatedStart != nullptr) { + lastIssueType = accessing_addr <= allocatedStart ? IssueType::UNDERFLOW : IssueType::OVERFLOW; + } else { + lastIssueType = IssueType::UNKNOWN; + } + } else { + lastIssueType = IssueType::USE_AFTER_FREE; + } + } + + int fd = -1; + if (!gOpts.issueDumpFilePath.empty()) { + fd = TEMP_FAILURE_RETRY(syscall(__NR_openat, AT_FDCWD, gOpts.issueDumpFilePath.c_str(), + O_RDWR | O_CREAT, paths::kDefaultDataFilePermission)); + if (fd < 0) { + int errcode = errno; + LOGE(LOG_TAG, "Error: %s(%d), fail to open file %s for dumping issue.", + strerror(errcode), errcode, gOpts.issueDumpFilePath.c_str()); + } + syscall(__NR_fchmod, fd, paths::kDefaultDataFilePermission); + } + ON_SCOPE_EXIT(if (fd >= 0) syscall(__NR_close, fd)); + + size_t allocatedSize = pagepool::GetAllocatedSize(slot); + pid_t accessingThreadId = gettid(); + issue::ThreadName accessingThreadName = {}; + issue::GetSelfThreadName(accessingThreadName); + void* accessingStackPCs[pagepool::MAX_RECORDED_STACKFRAME_COUNT] = {}; + size_t accessingStackElemCount = + unwind::UnwindStack(ucontext, accessingStackPCs, pagepool::MAX_RECORDED_STACKFRAME_COUNT); + auto accessingStackTrace = GetReadableStackTrace((const void**) accessingStackPCs, accessingStackElemCount); + + pid_t allocateThreadId = 0; + char* allocateThreadName = nullptr; + pagepool::GetThreadInfoOnBorrow(slot, &allocateThreadId, &allocateThreadName); + void** allocateStackPCs = nullptr; + size_t allocateStackElemCount = 0; + pagepool::GetStackTraceOnBorrow(slot, &allocateStackPCs, &allocateStackElemCount); + auto allocateStackTrace = GetReadableStackTrace((const void**) allocateStackPCs, allocateStackElemCount); + + pid_t freeThreadId = 0; + char* freeThreadName = nullptr; + void** freeStackPCs = nullptr; + size_t freeStackElemCount = 0; + std::vector freeStackTrace; + + bool printFreeTrace = lastIssueType == IssueType::DOUBLE_FREE || lastIssueType == IssueType::USE_AFTER_FREE; + if (printFreeTrace) { + pagepool::GetThreadInfoOnReturn(slot, &freeThreadId, &freeThreadName); + pagepool::GetStackTraceOnReturn(slot, &freeStackPCs, &freeStackElemCount); + freeStackTrace = GetReadableStackTrace((const void**) freeStackPCs, freeStackElemCount); + } + + char line[256] = {}; + PrintLine(fd, "*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***"); + PrintLine(fd, "Process: %s", GetCurrentProcessName(line, sizeof(line))); + PrintLine(fd, "Report Time: %s", GetReadableTimeStamp(line, sizeof(line))); + switch (lastIssueType) { + case IssueType::OVERFLOW: { + size_t overlappedRelOffset = (size_t) accessing_addr - (size_t) allocatedStart - allocatedSize; + PrintLine(fd, + "Issue: Overflow at %p (%" PRIuPTR " bytes after upper bound) of a %" PRIuPTR "-bytes buffer from thread %s(%d).", + accessing_addr, overlappedRelOffset, allocatedSize, accessingThreadName, accessingThreadId); + break; + } + case IssueType::UNDERFLOW: { + size_t overlappedRelOffset = (size_t) allocatedStart - (size_t) accessing_addr - allocatedSize; + PrintLine(fd, + "Issue: Underflow at %p (%" PRIuPTR " bytes before lower bound) of a %" PRIuPTR "-bytes buffer from thread %s(%d).", + accessing_addr, overlappedRelOffset, allocatedSize, accessingThreadName, accessingThreadId); + break; + } + case IssueType::USE_AFTER_FREE: { + PrintLine(fd, + "Issue: Use after free on a %" PRIuPTR "-bytes buffer at %p from thread %s(%d).", + allocatedSize, accessing_addr, accessingThreadName, accessingThreadId); + break; + } + case IssueType::DOUBLE_FREE: { + PrintLine(fd, + "Issue: Double free on a %" PRIuPTR "-bytes buffer at %p from thread %s(%d).", + allocatedSize, accessing_addr, accessingThreadName, accessingThreadId); + break; + } + case IssueType::UNKNOWN: + default: { + PrintLine(fd, + "Issue: Unknown failure when access a %" PRIuPTR "-bytes buffer at %p from thread %s(%d).", + allocatedSize, accessing_addr, accessingThreadName, accessingThreadId); + break; + } + } + PrintStackTrace(fd, accessingStackTrace, "stacktrace:"); + if (printFreeTrace) { + PrintStackTrace(fd, freeStackTrace, + "free by thread %s(%d):", freeThreadName, freeThreadId); + } + PrintStackTrace(fd, allocateStackTrace, + "allocate by thread %s(%d):", allocateThreadName, allocateThreadId); + + return true; +} + +const char* memguard::issue::GetIssueTypeName(memguard::IssueType type) { + switch (type) { + case IssueType::OVERFLOW: return "OVERFLOW"; + case IssueType::UNDERFLOW: return "UNDERFLOW"; + case IssueType::USE_AFTER_FREE: return "USE_AFTER_FREE"; + case IssueType::DOUBLE_FREE: return "DOUBLE_FREE"; + case IssueType::UNKNOWN: default: return "UNKNOWN"; + } +} + +void memguard::issue::GetSelfThreadName(ThreadName name_out) { + if (syscall(__NR_prctl, PR_GET_NAME, name_out, 0, 0, 0) != 0) { + name_out[0] = '\0'; + } else { + name_out[sizeof(ThreadName) / sizeof(char) - 1] = '\0'; + } +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Issue.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Issue.h new file mode 100644 index 000000000..15e2e4cc4 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Issue.h @@ -0,0 +1,42 @@ +// +// Created by tomystang on 2020/11/24. +// + +#ifndef __MEMGUARD_ISSUE_H__ +#define __MEMGUARD_ISSUE_H__ + + +#include +#include "Auxiliary.h" +#include "PagePool.h" + +// This value is not exported by kernel headers. +// DoNot change this value since it's related to kernel limitations. It has not +// help to expand or truncate fetched thread name by changing this value. +#define MAX_TASK_COMM_LEN 16 + +namespace memguard { + enum IssueType { + UNKNOWN = -1, + OVERFLOW = 1, + UNDERFLOW = 2, + USE_AFTER_FREE = 3, + DOUBLE_FREE = 4, + }; + + namespace issue { + extern IssueType TLSVAR gLastIssueType; + + extern void TriggerIssue(pagepool::slot_t accessing_slot, IssueType type); + extern IssueType GetLastIssueType(); + + extern bool Report(const void* accessing_addr, void* ucontext); + extern const char* GetIssueTypeName(IssueType type); + + typedef char ThreadName[MAX_TASK_COMM_LEN]; + extern void GetSelfThreadName(ThreadName name_out); + } +} + + +#endif //__MEMGUARD_ISSUE_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Log.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Log.h new file mode 100644 index 000000000..6d44afa89 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Log.h @@ -0,0 +1,93 @@ +// +// Created by tomystang on 2020/11/24. +// + +#ifndef __MEMGUARD_LOG_H__ +#define __MEMGUARD_LOG_H__ + + +#include +#include "Auxiliary.h" + +#ifndef MEMGUARD_LOG_LEVEL +#define MEMGUARD_LOG_LEVEL LOG_DEBUG +#endif + +#define LOG_VERBOSE 2 +#define LOG_DEBUG 3 +#define LOG_INFO 4 +#define LOG_WARN 5 +#define LOG_ERROR 6 +#define LOG_FATAL 7 +#define LOG_SILENT 8 + +#define PRINT_LOG(level, tag, fmt, args...) do { memguard::log::PrintLog(level, tag, fmt, ##args); } while (false) +#define PRINT_LOG_V(level, tag, fmt, va_args) do { memguard::log::PrintLogV(level, tag, fmt, va_args); } while (false) + +static inline void OmitLog(const char* tag, const char* fmt, ...) { UNUSED(tag); UNUSED(fmt); } +static inline void OmitLogV(const char* tag, const char* fmt, va_list va_args) { UNUSED(tag); UNUSED(fmt); UNUSED(va_args); } + +#if MEMGUARD_LOG_LEVEL <= LOG_VERBOSE +#define LOGV(tag, fmt, args...) PRINT_LOG(ANDROID_LOG_VERBOSE, tag, fmt, ##args) +#define LOGV_V(tag, fmt, va_args) PRINT_LOG_V(ANDROID_LOG_VERBOSE, tag, fmt, va_args) +#else +#define LOGV(tag, fmt, args...) OmitLog(tag, fmt, ##args) +#define LOGV_V(tag, fmt, va_args) OmitLogV(tag, fmt, va_args) +#endif + +#if MEMGUARD_LOG_LEVEL <= LOG_DEBUG +#define LOGD(tag, fmt, args...) PRINT_LOG(ANDROID_LOG_DEBUG, tag, fmt, ##args) +#define LOGD_V(tag, fmt, va_args) PRINT_LOG_V(ANDROID_LOG_DEBUG, tag, fmt, va_args) +#else +#define LOGD(tag, fmt, args...) OmitLog(tag, fmt, ##args) +#define LOGD_V(tag, fmt, va_args) OmitLogV(tag, fmt, va_args) +#endif + +#if MEMGUARD_LOG_LEVEL <= LOG_INFO +#define LOGI(tag, fmt, args...) PRINT_LOG(ANDROID_LOG_INFO, tag, fmt, ##args) +#define LOGI_V(tag, fmt, va_args) PRINT_LOG_V(ANDROID_LOG_INFO, tag, fmt, va_args) +#else +#define LOGI(tag, fmt, args...) OmitLog(tag, fmt, ##args) +#define LOGI_V(tag, fmt, va_args) OmitLogV(tag, fmt, va_args) +#endif + +#if MEMGUARD_LOG_LEVEL <= LOG_WARN +#define LOGW(tag, fmt, args...) PRINT_LOG(ANDROID_LOG_WARN, tag, fmt, ##args) +#define LOGW_V(tag, fmt, va_args) PRINT_LOG_V(ANDROID_LOG_WARN, tag, fmt, va_args) +#else +#define LOGW(tag, fmt, args...) OmitLog(tag, fmt, ##args) +#define LOGW_V(tag, fmt, va_args) OmitLogV(tag, fmt, va_args) +#endif + +#if MEMGUARD_LOG_LEVEL <= LOG_ERROR +#define LOGE(tag, fmt, args...) PRINT_LOG(ANDROID_LOG_ERROR, tag, fmt, ##args) +#define LOGE_V(tag, fmt, va_args) PRINT_LOG_V(ANDROID_LOG_ERROR, tag, fmt, va_args) +#else +#define LOGE(tag, fmt, args...) OmitLog(tag, fmt, ##args) +#define LOGE_V(tag, fmt, va_args) OmitLogV(tag, fmt, va_args) +#endif + +#if MEMGUARD_LOG_LEVEL <= LOG_FATAL +#define LOGF(tag, fmt, args...) PRINT_LOG(ANDROID_LOG_FATAL, tag, fmt, ##args) +#define LOGF_V(tag, fmt, va_args) PRINT_LOG_V(ANDROID_LOG_FATAL, tag, fmt, va_args) +#else +#define LOGF(tag, fmt, args...) OmitLog(tag, fmt, ##args) +#define LOGF_V(tag, fmt, va_args) OmitLogV(tag, fmt, va_args) +#endif + +#define ASSERT(cond, fmt, args...) \ + do { \ + if (UNLIKELY(!(cond))) { \ + __android_log_assert(nullptr, "MemGuard", "Assertion failed: %s, msg: " fmt, #cond, ##args); \ + } \ + } while (0) + +namespace memguard { + namespace log { + extern void PrintLog(int level, const char* tag, const char* fmt, ...); + extern void PrintLogV(int level, const char* tag, const char* fmt, va_list args); + } +} + + +#endif //__MEMGUARD_LOG_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Memory.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Memory.h new file mode 100644 index 000000000..6e5c66867 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Memory.h @@ -0,0 +1,22 @@ +// +// Created by tomystang on 2020/11/26. +// + +#ifndef __MEMGUARD_MEMORY_H__ +#define __MEMGUARD_MEMORY_H__ + + +#include + +namespace memguard { + namespace memory { + extern void* MapMemArea(size_t size, const char* region_name, void* hint = nullptr, bool force_hint = false); + extern void UnmapMemArea(void* addr, size_t size); + extern void MarkAreaReadWrite(void* start, size_t size); + extern void MarkAreaInaccessible(void* start, size_t size); + extern size_t GetPageSize(); + } +} + + +#endif //__MEMGUARD_MEMORY_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Mutex.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Mutex.h new file mode 100644 index 000000000..c61843155 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Mutex.h @@ -0,0 +1,34 @@ +// +// Created by tomystang on 2020/11/26. +// + +#ifndef __MEMGUARD_MUTEX_H__ +#define __MEMGUARD_MUTEX_H__ + + +#include +#include +#include + +namespace memguard { + class Mutex { + public: + Mutex(): mRawMutex(PTHREAD_MUTEX_INITIALIZER) {} + + bool tryLock(); + + void lock(); + + void unlock(); + + private: + DISALLOW_COPY(Mutex); + DISALLOW_MOVE(Mutex); + DISALLOW_NEW(Mutex); + + pthread_mutex_t mRawMutex; + }; +} + + +#endif //__MEMGUARD_MUTEX_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/PagePool.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/PagePool.cpp new file mode 100644 index 000000000..2cf0b882e --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/PagePool.cpp @@ -0,0 +1,287 @@ +// +// Created by tomystang on 2020/10/16. +// + +#include +#include +#include +#include +#include +#include "Auxiliary.h" +#include "PagePool.h" +#include "Memory.h" +#include "Mutex.h" +#include "Log.h" +#include "Issue.h" +#include + +using namespace memguard; + +#define LOG_TAG "MemGuard.PagePool" + +#define SLOT_AREA_TAG "MemGuard.Slot" +#define META_AREA_TAG "MemGuard.Meta" +#define FREE_SLOT_ID_AREA_TAG "MemGuard.FreeSlotID" + +enum InternalGuardSide { + UNINITIALIZED = 0, + ON_LEFT = 1, + ON_RIGHT = 2 +}; + +struct alignas(8) PageMeta { + bool isBorrowed = false; + void* allocatedStart = nullptr; + size_t allocatedSize = 0; + InternalGuardSide guardSide = UNINITIALIZED; + pid_t borrowThreadId = 0; + issue::ThreadName borrowThreadName; + void* stackTraceOnBorrow[pagepool::MAX_RECORDED_STACKFRAME_COUNT]; + size_t onBorrowStackFrameCount = 0; + pid_t returnThreadId = 0; + issue::ThreadName returnThreadName; + void* stackTraceOnReturn[pagepool::MAX_RECORDED_STACKFRAME_COUNT]; + size_t onReturnStackFrameCount = 0; +}; + +static Mutex sPoolMutex; + +static size_t sGuardPageSize = 0; +static size_t sSlotSize = 0; +static void* sSlotAreaStart = nullptr; +static size_t sSlotAreaSize = 0; +static void* sSlotAreaEnd = nullptr; + +static PageMeta* sMetaAreaStart = nullptr; + +static pagepool::slot_t* sFreeSlots = nullptr; +static size_t sFreeSlotCount = 0; + +static size_t sLentPageCount = 0; + +bool memguard::pagepool::Prepare() { + sGuardPageSize = memory::GetPageSize(); + sSlotSize = AlignUpTo(gOpts.maxAllocationSize, memory::GetPageSize()); + sSlotAreaSize = sGuardPageSize + (sSlotSize + sGuardPageSize) * gOpts.maxDetectableAllocationCount; + sSlotAreaStart = memory::MapMemArea(sSlotAreaSize, SLOT_AREA_TAG); + if (sSlotAreaStart == nullptr) { + int errcode = errno; + LOGE(LOG_TAG, "Fail to mmap data area, error: %s(%d)", strerror(errcode), errcode); + return false; + } + sSlotAreaEnd = (void*) (((uint64_t) sSlotAreaStart) + sSlotAreaSize); + + size_t metaAreaSize = sizeof(PageMeta) * gOpts.maxDetectableAllocationCount; + sMetaAreaStart = (PageMeta*) memory::MapMemArea(metaAreaSize, META_AREA_TAG); + if (sMetaAreaStart == nullptr) { + int errcode = errno; + LOGE(LOG_TAG, "Fail to mmap meta area, error: %s(%d)", strerror(errcode), errcode); + return false; + } + memory::MarkAreaReadWrite(sMetaAreaStart, metaAreaSize); + memset(sMetaAreaStart, 0, metaAreaSize); + + size_t freeSlotsSize = sizeof(slot_t) * gOpts.maxDetectableAllocationCount; + sFreeSlots = (slot_t*) memory::MapMemArea(freeSlotsSize, FREE_SLOT_ID_AREA_TAG); + if (sFreeSlots == nullptr) { + int errcode = errno; + LOGE(LOG_TAG, "Fail to mmap slot id area, error: %s(%d)", strerror(errcode), errcode); + return false; + } + memory::MarkAreaReadWrite(sFreeSlots, freeSlotsSize); + memset(sFreeSlots, 0, freeSlotsSize); + + return true; +} + +int memguard::pagepool::BorrowSlot(size_t size, size_t alignment, GuardSide guard_side) { + if (size > sSlotSize) { + return SLOT_NONE; + } + if (AlignUpTo(size, alignment) > pagepool::GetSlotSize()) { + return SLOT_NONE; + } + + slot_t resultSlotID = SLOT_NONE; + { + sPoolMutex.lock(); + ON_SCOPE_EXIT(sPoolMutex.unlock()); + + if (sLentPageCount < gOpts.maxDetectableAllocationCount) { + resultSlotID = sLentPageCount++; + } else if (sFreeSlotCount > 0) { + int pickedSlotIndex = (int) (random::GenerateUnsignedInt32() % sFreeSlotCount); + resultSlotID = sFreeSlots[pickedSlotIndex]; + sFreeSlots[pickedSlotIndex] = sFreeSlots[--sFreeSlotCount]; + } + + if (resultSlotID == SLOT_NONE) { + return resultSlotID; + } + } + + auto meta = sMetaAreaStart + resultSlotID; + meta->isBorrowed = true; + meta->allocatedSize = size; + + meta->borrowThreadId = gettid(); + issue::GetSelfThreadName(meta->borrowThreadName); + meta->onBorrowStackFrameCount = unwind::UnwindStack( + meta->stackTraceOnBorrow, pagepool::MAX_RECORDED_STACKFRAME_COUNT); + + meta->returnThreadId = 0; + meta->returnThreadName[0] = '\0'; + meta->onReturnStackFrameCount = 0; + + off_t allocatedStart = 0; + if (guard_side == ON_RIGHT) { + uint64_t padding = pagepool::GetSlotSize() - size; + if (!gOpts.perfectRightSideGuard) { + padding = AlignDownTo(padding, alignment); + } + allocatedStart = padding; + } + void* guardedPageStart = GetGuardedPageStart(resultSlotID); + meta->allocatedStart = (void*) ((uint64_t) guardedPageStart + allocatedStart); + + void* pagedAllocatedStart = (void*) AlignDownTo((uint64_t) meta->allocatedStart, memory::GetPageSize()); + size_t pagedAllocatedSize = AlignUpTo(size, memory::GetPageSize()); + memory::MarkAreaReadWrite(pagedAllocatedStart, pagedAllocatedSize); + // memset(pagedAllocatedStart, 0xAD, pagedAllocatedSize); + + return resultSlotID; +} + +bool memguard::pagepool::IsSlotBorrowed(slot_t slot) { + ASSERT(slot >= 0 && (unsigned) slot < gOpts.maxDetectableAllocationCount, "Bad slot: %" PRId64, slot); + return sMetaAreaStart[slot].isBorrowed; +} + +void memguard::pagepool::ReturnSlot(slot_t slot) { + ASSERT(slot >= 0 && (unsigned) slot < gOpts.maxDetectableAllocationCount, "Bad slot: %" PRId64, slot); + + auto meta = sMetaAreaStart + slot; + if (!meta->isBorrowed) { + LOGE(LOG_TAG, "Double free detected."); + issue::TriggerIssue(slot, IssueType::DOUBLE_FREE); + } + + meta->isBorrowed = false; + + meta->returnThreadId = gettid(); + issue::GetSelfThreadName(meta->returnThreadName); + meta->onReturnStackFrameCount = + unwind::UnwindStack(meta->stackTraceOnReturn, pagepool::MAX_RECORDED_STACKFRAME_COUNT); + + void* guardedPageStart = GetGuardedPageStart(slot); + memory::MarkAreaReadWrite(guardedPageStart, GetSlotSize()); + // memset(guardedPageStart, 0xBF, GetSlotSize()); + memory::MarkAreaInaccessible(guardedPageStart, GetSlotSize()); + + { + sPoolMutex.lock(); + ON_SCOPE_EXIT(sPoolMutex.unlock()); + + if (sFreeSlotCount >= gOpts.maxDetectableAllocationCount) { + return; + } + + sFreeSlots[sFreeSlotCount++] = slot; + } +} + +/* + * Slot Area Layout + * +-----+------------+-----+------------+-----+ + * | | | | | | + * | | Guarded | | Guarded | | + * |Guard| Pages |Guard| Pages |Guard| + * | | | | | | + * +------------------+------------------+-----+ + * | | | | + * +-+ slot 0 +-+ +-+ slot 1 +-+ + * + */ +void* memguard::pagepool::GetGuardedPageStart(slot_t slot) { + ASSERT(slot >= 0 && (unsigned) slot < gOpts.maxDetectableAllocationCount, "Bad slot: %" PRId64, slot); + return (void*) ((uint64_t) (sSlotAreaStart) + sGuardPageSize + (sSlotSize + sGuardPageSize) * slot); +} + +void* memguard::pagepool::GetAllocatedAddress(slot_t slot) { + ASSERT(slot >= 0 && (unsigned) slot < gOpts.maxDetectableAllocationCount, "Bad slot: %" PRId64, slot); + return sMetaAreaStart[slot].allocatedStart; +} + +size_t memguard::pagepool::GetAllocatedSize(slot_t slot) { + ASSERT(slot >= 0 && (unsigned) slot < gOpts.maxDetectableAllocationCount, "Bad slot: %" PRId64, slot); + + sPoolMutex.lock(); + ON_SCOPE_EXIT(sPoolMutex.unlock()); + + return sMetaAreaStart[slot].allocatedSize; +} + +bool memguard::pagepool::GetSlotGuardSide(slot_t slot, GuardSide* result) { + if (slot == SLOT_NONE || result == nullptr) { + return false; + } + auto meta = sMetaAreaStart + slot; + if (!meta->isBorrowed) { + return false; + } + *result = meta->guardSide == InternalGuardSide::ON_LEFT ? GuardSide::ON_LEFT : GuardSide::ON_RIGHT; + return true; +} + +size_t memguard::pagepool::GetSlotSize() { + return sSlotSize; +} + +pagepool::slot_t memguard::pagepool::GetSlotIdOfAddress(const void* addr) { + if (addr < sSlotAreaStart || addr >= sSlotAreaEnd) { + return SLOT_NONE; + } + void* firstGuardedPageStart = GetGuardedPageStart(0); + if (addr < firstGuardedPageStart) { + return 0; + } + uint64_t distance = (uint64_t) addr - (uint64_t) firstGuardedPageStart; + return (slot_t) (distance / (sSlotSize + sGuardPageSize)); +} + +bool memguard::pagepool::IsAddressInGuardPage(const void* addr) { + if (addr < sSlotAreaStart || addr >= sSlotAreaEnd) { + return false; + } + size_t addrPageCount = ((uint64_t) addr - (uint64_t) sSlotAreaStart) / memory::GetPageSize(); + size_t maxAllocationPageCount = sSlotSize / memory::GetPageSize(); + return (addrPageCount % (maxAllocationPageCount + 1)) == 0; +} + +void memguard::pagepool::GetThreadInfoOnBorrow(slot_t slot, pid_t* tid_out, char** name_out) { + ASSERT(slot >= 0 && (unsigned) slot < gOpts.maxDetectableAllocationCount, "Bad slot: %" PRId64, slot); + auto meta = sMetaAreaStart + slot; + *tid_out = meta->borrowThreadId; + *name_out = meta->borrowThreadName; +} + +void memguard::pagepool::GetStackTraceOnBorrow(slot_t slot, void** pcs[], size_t* size) { + ASSERT(slot >= 0 && (unsigned) slot < gOpts.maxDetectableAllocationCount, "Bad slot: %" PRId64, slot); + auto meta = sMetaAreaStart + slot; + *pcs = meta->stackTraceOnBorrow; + *size = meta->onBorrowStackFrameCount; +} + +void memguard::pagepool::GetThreadInfoOnReturn(slot_t slot, pid_t* tid_out, char** name_out) { + ASSERT(slot >= 0 && (unsigned) slot < gOpts.maxDetectableAllocationCount, "Bad slot: %" PRId64, slot); + auto meta = sMetaAreaStart + slot; + *tid_out = meta->returnThreadId; + *name_out = meta->returnThreadName; +} + +void memguard::pagepool::GetStackTraceOnReturn(memguard::pagepool::slot_t slot, void** pcs[], size_t* size) { + ASSERT(slot >= 0 && (unsigned) slot < gOpts.maxDetectableAllocationCount, "Bad slot: %" PRId64, slot); + auto meta = sMetaAreaStart + slot; + *pcs = meta->stackTraceOnReturn; + *size = meta->onReturnStackFrameCount; +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/PagePool.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/PagePool.h new file mode 100644 index 000000000..34cbb60be --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/PagePool.h @@ -0,0 +1,47 @@ +// +// Created by tomystang on 2020/10/16. +// + +#ifndef __MEMGUARD_PAGEPOOL_H__ +#define __MEMGUARD_PAGEPOOL_H__ + + +#include + +namespace memguard { + namespace pagepool { + static constexpr const size_t MAX_RECORDED_STACKFRAME_COUNT = 16; + + typedef int64_t slot_t; + + static constexpr const slot_t SLOT_NONE = -1; + + enum GuardSide { + ON_LEFT = 1, + ON_RIGHT = 2 + }; + + extern bool Prepare(); + + extern int BorrowSlot(size_t size, size_t alignment, GuardSide guard_side); + extern bool IsSlotBorrowed(slot_t slot); + extern void ReturnSlot(slot_t slot); + + extern void* GetGuardedPageStart(slot_t slot); + extern void* GetAllocatedAddress(slot_t slot); + extern size_t GetAllocatedSize(slot_t slot); + extern bool GetSlotGuardSide(slot_t slot, GuardSide* result); + extern size_t GetSlotSize(); + + extern slot_t GetSlotIdOfAddress(const void* addr); + extern bool IsAddressInGuardPage(const void* addr); + + extern void GetThreadInfoOnBorrow(slot_t slot, pid_t* tid_out, char** name_out); + extern void GetStackTraceOnBorrow(slot_t slot, void** pcs[], size_t* size); + extern void GetThreadInfoOnReturn(slot_t slot, pid_t* tid_out, char** name_out); + extern void GetStackTraceOnReturn(slot_t slot, void** pcs[], size_t* size); + } +} + + +#endif //__MEMGUARD_PAGEPOOL_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Paths.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Paths.h new file mode 100644 index 000000000..1397310df --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Paths.h @@ -0,0 +1,24 @@ +// +// Created by tomystang on 2021/2/2. +// + +#ifndef __MEMGUARD_PATHS_H__ +#define __MEMGUARD_PATHS_H__ + + +#include +#include + +namespace memguard { + namespace paths { + constexpr const int kDefaultDirectoryPermission = 00775; + constexpr const int kDefaultDataFilePermission = 00664; + + extern bool Exists(const std::string& path); + extern std::string GetParent(const std::string& path); + extern bool MakeDirs(const std::string& path, int mode = kDefaultDirectoryPermission); + } +} + + +#endif //__MEMGUARD_PATHS_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/SignalHandler.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/SignalHandler.cpp new file mode 100644 index 000000000..d3f1721b6 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/SignalHandler.cpp @@ -0,0 +1,90 @@ +// +// Created by tomystang on 2020/10/16. +// + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "Allocation.h" + +using namespace memguard; + +#define LOG_TAG "MemGuard.SignalHandler" + +static struct sigaction sPrevSigAction = {}; + +struct RoutineParams { + siginfo_t* info; + ucontext_t* ucontext; +}; + +static int ReportRoutine(void* param) { + auto actualParams = (RoutineParams*) param; + return issue::Report(actualParams->info->si_addr, actualParams->ucontext) ? 0 : -1; +} + +static int NotifyDumpedIssueRoutine(void* param) { + UNUSED(param); + pthread_t hThread; + pthread_create(&hThread, nullptr, [](void*) -> void* { + c2j::NotifyOnIssueDumpped(gOpts.issueDumpFilePath.c_str()); + return nullptr; + }, nullptr); + pthread_join(hThread, nullptr); + return 0; +} + +static void SegvSignalHandler(int sig, siginfo_t* info, void* ucontext) { + if (info->si_code == SEGV_ACCERR && allocation::IsAllocatedByThisAllocator(info->si_addr)) { + Thread reportThread(64 * 1024, Thread::FixStackForART, ucontext); + RoutineParams params = {.info = info, .ucontext = (ucontext_t*) ucontext}; + int result = 0; + int error = reportThread.startAndWait(ReportRoutine, ¶ms, &result); + LOGD(LOG_TAG, "ReportRoutine called, result: %d, error: %d", result, error); + if (error == Thread::RESULT_SUCCESS) { + error = reportThread.startAndWait(NotifyDumpedIssueRoutine, ¶ms, &result); + LOGD(LOG_TAG, "NotifyDumpedIssueRoutine called, result: %d, error: %d", result, error); + } + } + + // Process any previous handlers. + if (((unsigned) sPrevSigAction.sa_flags & SA_SIGINFO) != 0) { + sPrevSigAction.sa_sigaction(sig, info, ucontext); + } else if (sPrevSigAction.sa_handler == SIG_DFL) { + // If the previous handler was the default handler, cause a core dump. + signal(SIGSEGV, SIG_DFL); + raise(SIGSEGV); + } else if (sPrevSigAction.sa_handler == SIG_IGN) { + // If the previous segv handler was SIGIGN, crash if and only if we were responsible + // for the crash. + if ((issue::GetLastIssueType() != IssueType::UNKNOWN && info->si_addr != nullptr) + || (allocation::IsAllocatedByThisAllocator(info->si_addr))) { + signal(SIGSEGV, SIG_DFL); + raise(SIGSEGV); + } + } else { + sPrevSigAction.sa_handler(sig); + } +} + +bool memguard::signalhandler::Install() { + struct sigaction sigAction = {}; + sigAction.sa_sigaction = SegvSignalHandler; + sigAction.sa_flags = SA_SIGINFO; + if (::sigaction(SIGSEGV, &sigAction, &sPrevSigAction) == -1) { + int errcode = errno; + LOGE(LOG_TAG, "Fail to call sigaction, error: %d (%s)", errcode, strerror(errcode)); + return false; + } + LOGD(LOG_TAG, "SignalHandler was installed."); + return true; +} \ No newline at end of file diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/SignalHandler.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/SignalHandler.h new file mode 100644 index 000000000..1ae164f05 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/SignalHandler.h @@ -0,0 +1,16 @@ +// +// Created by tomystang on 2020/10/16. +// + +#ifndef __MEMGUARD_SIGNALHANDLER_H__ +#define __MEMGUARD_SIGNALHANDLER_H__ + + +namespace memguard { + namespace signalhandler { + bool Install(); + } +} + + +#endif //__MEMGUARD_SIGNALHANDLER_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Thread.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Thread.cpp new file mode 100644 index 000000000..f1203c45f --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Thread.cpp @@ -0,0 +1,246 @@ +// +// Created by tomystang on 2020/11/27. +// + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "Log.h" +#include "Memory.h" +#include "Thread.h" +#include "FdSanWrapper.h" + +using namespace memguard; + +#define LOG_TAG "MemGuard.Thread" +#define STACK_REGION_TAG "MemGuard.ThreadStack" + +static bool IterateSelfMaps(void* buffer, size_t buffer_size, const std::function& callback) { + if (buffer == nullptr || buffer_size == 0) { + LOGE(LOG_TAG, "buffer is null or buffer_size is zero."); + return false; + } + + int fd = TEMP_FAILURE_RETRY(syscall(__NR_openat, AT_FDCWD, "/proc/self/maps", O_RDONLY | O_CLOEXEC)); + if (fd == -1) { + int errcode = errno; + LOGE(LOG_TAG, "fail to open /proc/self/maps, error: %s(%d)", strerror(errcode), errcode); + return false; + } + ON_SCOPE_EXIT(syscall(__NR_close, fd)); + + char* charBuffer = (char*) buffer; + size_t start = 0; + size_t bytesRead = 0; + bool readComplete = false; + while (true) { + ssize_t currBytesRead = TEMP_FAILURE_RETRY(syscall(__NR_read, fd, charBuffer + bytesRead, buffer_size - bytesRead - 1)); + if (currBytesRead <= 0) { + if (bytesRead == 0) { + return currBytesRead == 0; + } + // Treat the last piece of data as the last line. + charBuffer[start + bytesRead] = '\n'; + currBytesRead = 1; + readComplete = true; + } + bytesRead += currBytesRead; + + while (bytesRead > 0) { + char* newline = (char*) memchr(charBuffer + start, '\n', bytesRead); + if (newline == nullptr) { + break; + } + *newline = '\0'; + char* line = charBuffer + start; + start = newline - charBuffer + 1; + size_t lineLength = newline - line + 1; + bytesRead -= lineLength; + + char* firstBarPos = (char*) memchr(line, '-', lineLength); + if (firstBarPos != nullptr) { + *firstBarPos = '\0'; + uintptr_t startAddr = strtoll(line, nullptr, 16); + uintptr_t endAddr = strtoll(firstBarPos + 1, nullptr, 16); + if (callback(startAddr, endAddr)) { + return true; + } + } + } + + if (readComplete) { + return true; + } + + if (start == 0 && bytesRead == buffer_size - 1) { + LOGE(LOG_TAG, "buffer is too small to hold a single line."); + return false; + } + + // Copy any leftover data to the front of the buffer. + if (start > 0) { + if (bytesRead > 0) { + memmove(charBuffer, charBuffer + start, bytesRead); + } + start = 0; + } + } +} + +// Tricky but probably the only way to find a place for allocating thread stack that +// is satisfied with the stack base limitation of ART runtime. +// Credit to johnwhe. +static std::pair SearchStackHintForARTStackFixing(size_t stack_size, void* ucontext) { + uintptr_t sp = 0; + #if defined (__arm__) + sp = ((ucontext_t*) ucontext)->uc_mcontext.arm_sp; + #elif defined (__aarch64__) + sp = ((ucontext_t*) ucontext)->uc_mcontext.sp; + #elif defined (__i386__) + sp = ((ucontext_t*) ucontext)->uc_mcontext.gregs[REG_ESP]; + #elif defined (__x86_64__) + sp = ((ucontext_t*) ucontext)->uc_mcontext.gregs[REG_RSP]; + #endif + if (UNLIKELY(sp == 0)) { + LOGE(LOG_TAG, "Fail to get sp from ucontext, return empty hint."); + return std::make_pair(nullptr, false); + } + + // TODO: constructing std::function by lambda may trigger heap allocation, and may cause + // problems in signal handler. + struct { + size_t stackSize; + uintptr_t sp; + uintptr_t lastEnd; + uintptr_t result; + } args {stack_size, sp, 0, 0}; + char lineBuffer[1024]; + auto iterCb = [&args](uintptr_t startAddr, uintptr_t endAddr) { + if (args.sp < args.lastEnd && startAddr - args.lastEnd >= args.stackSize) { + args.result = args.lastEnd; + return true; + } + args.lastEnd = endAddr; + return false; + }; + bool iterOK = IterateSelfMaps(lineBuffer, sizeof(lineBuffer), iterCb); + + if (iterOK && args.result == 0) { + // Check unused space after last mapped item. + uintptr_t maxAddr = (((uintptr_t) 1) << (sizeof(void*) * 8 - 1)); + if (maxAddr - args.lastEnd >= stack_size) { + args.result = args.lastEnd; + } + } + + void* finalResult = nullptr; + bool force; + if (!iterOK || args.result == 0) { + finalResult = (void*) AlignUpTo(sp + stack_size, memory::GetPageSize()); + force = false; + } else { + finalResult = (void*) args.result; + force = true; + } + + LOGI(LOG_TAG, "Found available stack address, sp: %p, found_addr: %p, force: %d", sp, finalResult, force); + return std::make_pair(finalResult, force); +} + +memguard::Thread::Thread(size_t stack_size, const struct Thread::FixStackForART&, void* ucontext) + : Thread(stack_size, SearchStackHintForARTStackFixing(stack_size, ucontext)) {} + +memguard::Thread::Thread(size_t stack_size, const std::pair& hint) { + mStackSize = 0; + + void* stackHint = hint.first; + bool forceHint = hint.second; + + mStack = memory::MapMemArea(stack_size, STACK_REGION_TAG, stackHint, forceHint); + if (LIKELY(mStack != nullptr)) { + LOGD(LOG_TAG, "Stack mapped, size: %" PRIu32 ", stack_hint: %p, force_hint: %d, stack_addr: %p", + stack_size, stackHint, forceHint, mStack); + memory::MarkAreaReadWrite(mStack, stack_size); + mStackSize = stack_size; + } else if (forceHint) { + LOGE(LOG_TAG, "Fail to allocate %" PRIu32 " bytes stack with hint addr: %p, size: %" PRIu32, stack_size, stackHint); + } + if (mStack == nullptr) { + mStack = memory::MapMemArea(stack_size, STACK_REGION_TAG, stackHint, false); + if (LIKELY(mStack != nullptr)) { + LOGD(LOG_TAG, "Stack mapped, size: %" PRIu32 ", stack_hint: %p, force_hint: %d, stack_addr: %p", + stack_size, stackHint, forceHint, mStack); + memory::MarkAreaReadWrite(mStack, stack_size); + mStackSize = stack_size; + } else { + LOGE(LOG_TAG, "Fail to allocate %" PRIu32 " bytes stack again with hint addr: %p, size: %" PRIu32, stack_size, stackHint); + } + } +} + +struct Context { + memguard::Thread::RoutineFunction routine_function; + void* param; + pid_t tid; + int return_value; + + explicit Context(memguard::Thread::RoutineFunction routine_function, void* param) + : routine_function(routine_function), param(param), tid(-1), return_value(0) {} +}; + +static int ThreadInternalRoutine(void* param) { + auto origFDSanLevel = AndroidFdSanSetErrorLevel(android_fdsan_error_level::ANDROID_FDSAN_ERROR_LEVEL_DISABLED); + Context* context = (Context*) param; + context->return_value = context->routine_function(context->param); + AndroidFdSanSetErrorLevel(origFDSanLevel); + return context->return_value; +} + +int memguard::Thread::startAndWait(Thread::RoutineFunction routine, void* param, int* return_value) { + if (mStack == nullptr) { + return RESULT_INITIALIZE_FAILED; + } + + int flags = CLONE_UNTRACED | CLONE_VM | CLONE_SIGHAND + | CLONE_THREAD | CLONE_PARENT_SETTID | CLONE_CHILD_CLEARTID; + + Context context(routine, param); + + pid_t child = clone(ThreadInternalRoutine, (void*) ((uint64_t) mStack + mStackSize), + flags, &context, &context.tid, nullptr, &context.tid); + if (child == -1) { + return RESULT_CLONE_FAILED; + } + + // Wait for thread starting. + while (context.tid) { + syscall(__NR_futex, &context.tid, FUTEX_WAIT, -1, nullptr, nullptr, 0); + } + + // Wait for thread stopping. + while (context.tid) { + // clone will unblock us from waiting on context.tid. + syscall(__NR_futex, &context.tid, FUTEX_WAIT, child, nullptr, nullptr, 0); + } + + if (return_value != nullptr) { + *return_value = context.return_value; + } + + return RESULT_SUCCESS; +} + +void memguard::Thread::destroy() { + if (mStack != nullptr) { + memory::UnmapMemArea(mStack, mStackSize); + mStackSize = 0; + } +} diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Thread.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Thread.h new file mode 100644 index 000000000..603d5bc95 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Thread.h @@ -0,0 +1,66 @@ +// +// Created by tomystang on 2020/11/27. +// + +#ifndef __MEMGUARD_THREAD_H__ +#define __MEMGUARD_THREAD_H__ + + +#include +#include +#include "Auxiliary.h" + +namespace memguard { + class Thread { + public: + enum : int { + RESULT_SUCCESS = 0, + RESULT_INITIALIZE_FAILED = -9001, + RESULT_CLONE_FAILED = -9002, + RESULT_WAIT_FAILED = -9003 + }; + + typedef int (*RoutineFunction)(void*); + + struct FixStackForART {}; + static inline constexpr const struct FixStackForART FixStackForART = {}; + + Thread(size_t stack_size, const struct FixStackForART&, void* ucontext); + + explicit Thread(size_t stack_size, void* stack_hint = nullptr, bool force_hint = false) + : Thread(stack_size, std::make_pair(stack_hint, force_hint)) {} + + Thread(Thread&& other) : mStack(other.mStack), mStackSize(other.mStackSize) { + other.mStack = nullptr; + other.mStackSize = 0; + } + + Thread& operator =(Thread&& rhs) { + destroy(); + mStack = rhs.mStack; + mStackSize = rhs.mStackSize; + rhs.mStack = nullptr; + rhs.mStackSize = 0; + return *this; + } + + virtual ~Thread() { + destroy(); + } + + int startAndWait(RoutineFunction routine, void* param, int* return_value); + + void destroy(); + + private: + DISALLOW_COPY(Thread); + + Thread(size_t stack_size, const std::pair& hint); + + void* mStack; + size_t mStackSize; + }; +} + + +#endif //__MEMGUARD_THREAD_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Unwind.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Unwind.h new file mode 100644 index 000000000..60dd08499 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memguard/util/Unwind.h @@ -0,0 +1,22 @@ +// +// Created by tomystang on 2020/10/15. +// + +#ifndef __MEMGUARD_UNWIND_H__ +#define __MEMGUARD_UNWIND_H__ + + +#include +#include + +namespace memguard { + namespace unwind { + extern bool Initialize(); + extern int UnwindStack(void** pcs, size_t max_count); + extern int UnwindStack(void* ucontext, void** pcs, size_t max_count); + extern char* GetStackElementDescription(const void* pc, char* desc, size_t max_length); + } +} + + +#endif //__MEMGUARD_UNWIND_H__ diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryBufferQueue.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryBufferQueue.cpp index d1ea5b16e..36f2c5e51 100644 --- a/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryBufferQueue.cpp +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryBufferQueue.cpp @@ -43,43 +43,20 @@ namespace matrix { HOOK_CHECK(assertion) } -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true - std::atomic g_queue_realloc_counter = 0; - std::atomic g_queue_realloc_size_counter = 0; - std::atomic g_queue_realloc_failure_counter = 0; - std::atomic g_queue_realloc_over_limit_counter = 0; - std::atomic g_queue_extra_stack_meta_allocated = 0; - std::atomic g_queue_extra_stack_meta_kept = 0; - -#else + std::atomic BufferQueueContainer::g_message_overflow_counter = 0; std::atomic BufferQueueContainer::g_locker_collision_counter = 0; - std::atomic BufferQueue::g_queue_realloc_memory_1_counter = 0; - std::atomic BufferQueue::g_queue_realloc_memory_2_counter = 0; - std::atomic BufferQueue::g_queue_realloc_reason_1_counter = 0; - std::atomic BufferQueue::g_queue_realloc_reason_2_counter = 0; - std::atomic BufferQueue::g_queue_realloc_failure_counter = 0; - std::atomic BufferQueue::g_queue_realloc_over_limit_counter = 0; + message_queue_counter_t BufferQueue::g_message_queue_counter_{}; + message_queue_counter_t BufferQueue::g_allocation_queue_counter_{}; + std::atomic BufferQueue::g_queue_extra_stack_meta_allocated = 0; std::atomic BufferQueue::g_queue_extra_stack_meta_kept = 0; -#endif BufferManagement::BufferManagement(memory_meta_container *memory_meta_container) { -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true - const size_t max_fold_ = 512; - - message_allocator_ = new ResizableFreeList(); - alloc_message_allocator_ = new ResizableFreeList(); - node_allocator_ = new ResizableFreeList(); -#endif - containers_.reserve(MAX_PTR_SLOT); for (int i = 0; i < MAX_PTR_SLOT; ++i) { auto container = new BufferQueueContainer(); -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true - container->queue_ = new BufferQueue(node_allocator_); -#endif containers_.emplace_back(container); } memory_meta_container_ = memory_meta_container; @@ -91,28 +68,21 @@ namespace matrix { delete container; } -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true - delete message_allocator_; - delete alloc_message_allocator_; - delete node_allocator_; -#else delete queue_swapped_; -#endif } [[noreturn]] void BufferManagement::process_routine(BufferManagement *this_) { + size_t last_total_message_counter = 0; + size_t total_message_counter = 0; while (true) { HOOK_LOG_ERROR("Process routine outside ... this_->containers_ %zu", this_->containers_.size()); -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE != true if (!this_->queue_swapped_) this_->queue_swapped_ = new BufferQueue(SIZE_AUGMENT); -#endif size_t busy_queue = 0; for (auto container : this_->containers_) { HOOK_LOG_ERROR("Process routine ... "); -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE != true BufferQueue *swapped = nullptr; { std::lock_guard lock(container->mutex_); @@ -130,36 +100,19 @@ namespace matrix { HOOK_LOG_ERROR("Swapped ... "); swapped->process( [&](message_t *message, allocation_message_t *allocation_message) { -#else - size_t message_counter = 0; - if (container) { - container->queue_->process( - [&](Node *message_node, Node *allocation_message_node) { - - message_t *message = &message_node->t_; - allocation_message_t *allocation_message = nullptr; - if (allocation_message_node) { - allocation_message = &allocation_message_node->t_; - } -#endif - if (message->type == message_type_allocation || + + total_message_counter++; + if (message->type == message_type_allocation || message->type == message_type_reallocation || message->type == message_type_mmap) { -#if USE_CRITICAL_CHECK == true - HOOK_CHECK(allocation_message); -#else - if (UNLIKELY(allocation_message == nullptr)) return; -#endif - -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true - message_counter++; - if (message_counter >= 5) { - busy_queue++; + + if (UNLIKELY(allocation_message == nullptr)) { + CRITICAL_CHECK(allocation_message); + return; } -#endif uint64_t stack_hash = 0; if (allocation_message->size != 0 && - allocation_message->backtrace.frame_size != 0) { + allocation_message->backtrace.frame_size != 0) { stack_hash = hash_frames( allocation_message->backtrace.frames, allocation_message->backtrace.frame_size); @@ -168,26 +121,17 @@ namespace matrix { this_->memory_meta_container_->insert( reinterpret_cast(allocation_message->ptr), stack_hash, -#if USE_SPLAY_MAP_SAVE_STACK == true allocation_message, -#endif [&](ptr_meta_t *ptr_meta, stack_meta_t *stack_meta) { ptr_meta->ptr = reinterpret_cast(allocation_message->ptr); ptr_meta->size = allocation_message->size; ptr_meta->attr.is_mmap = message->type == message_type_mmap; -#if USE_STACK_HASH_NO_COLLISION == true if (UNLIKELY(!stack_meta)) { ptr_meta->caller = allocation_message->caller; return; } -#else - ptr_meta->caller = allocation_message->caller; - if (UNLIKELY(!stack_meta)) { - return; - } -#endif stack_meta->size += allocation_message->size; if (stack_meta->backtrace.frame_size == 0 && @@ -201,22 +145,17 @@ namespace matrix { this_->memory_meta_container_->erase( reinterpret_cast(message->ptr)); } -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true - if (message_node) { - this_->message_allocator_->deallocate(message_node); - } - if (allocation_message_node) { - this_->alloc_message_allocator_->deallocate(allocation_message_node); - } - }); - if (message_counter > 0) { - HOOK_LOG_ERROR("Processed ... %zu messages ", message_counter); - } -#else }); swapped->reset(); this_->queue_swapped_ = swapped; -#endif + + if (total_message_counter - last_total_message_counter > 100000) { + HOOK_LOG_ERROR( + "Total Processed ... %zu messages, offer overflow counter %zu", + total_message_counter, + BufferQueueContainer::g_message_overflow_counter.load()); + last_total_message_counter = total_message_counter; + } } } @@ -233,6 +172,7 @@ namespace matrix { } else { usleep(PROCESS_IDLE_INTERVAL); } + } } diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryBufferQueue.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryBufferQueue.h index a98d1e9d3..60c061028 100644 --- a/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryBufferQueue.h +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryBufferQueue.h @@ -22,7 +22,6 @@ #include #include #include -#include class memory_meta_container; @@ -36,9 +35,10 @@ namespace matrix { typedef enum : uint8_t { message_type_nil = 0, message_type_allocation = 1, - message_type_deletion = 2, - message_type_mmap = 3, - message_type_munmap = 4 + message_type_reallocation = 2, + message_type_deletion = 3, + message_type_mmap = 4, + message_type_munmap = 5 } message_type; typedef wechat_backtrace::BacktraceFixed memory_backtrace_t; @@ -72,251 +72,196 @@ namespace matrix { return static_cast((_key ^ (_key >> 16)) & PTR_MASK); } -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE != true - class BufferQueue { - + typedef struct { + // Statistic + std::atomic realloc_memory_counter_; + std::atomic realloc_counter_; + std::atomic realloc_failure_counter_; + std::atomic realloc_reach_limit_counter_; + } message_queue_counter_t; + + template + class MessageAllocator { public: - BufferQueue(size_t size_augment) { - size_augment_ = size_augment; - buffer_realloc_message_queue(true); - buffer_realloc_alloc_queue(true); + MessageAllocator( + const size_t size_augment, + const size_t memory_limit, + message_queue_counter_t *const counter) + : size_augment_(size_augment), memory_limit_(memory_limit), counter_(counter) { + HOOK_CHECK(size_augment); + HOOK_CHECK(memory_limit); + HOOK_CHECK(counter); }; - ~BufferQueue() { - if (message_queue_) { - free(message_queue_); - message_queue_ = nullptr; - } - if (alloc_message_queue_) { - free(alloc_message_queue_); - alloc_message_queue_ = nullptr; - } + ~MessageAllocator() { + free(queue_); }; - inline allocation_message_t *enqueue_allocation_message(bool is_mmap) { - if (UNLIKELY(msg_queue_idx_ >= msg_queue_size_)) { - if (UNLIKELY(!buffer_realloc_message_queue(false))) { - return nullptr; - } - } - if (UNLIKELY(alloc_msg_queue_idx_ >= alloc_queue_size_)) { - if (UNLIKELY(!buffer_realloc_alloc_queue(false))) { - return nullptr; - } - } - - message_t *msg_idx = &message_queue_[msg_queue_idx_++]; - msg_idx->type = is_mmap ? message_type_mmap : message_type_allocation; - msg_idx->index = alloc_msg_queue_idx_; - - allocation_message_t *buffer = &alloc_message_queue_[alloc_msg_queue_idx_++]; - *buffer = {}; - return buffer; - } - - inline void enqueue_deletion_message(uintptr_t ptr, bool is_munmap) { + message_queue_counter_t *const counter_; - // Fast path. - if (msg_queue_idx_ > 0) { - message_t *msg_idx = &message_queue_[msg_queue_idx_ - 1]; - if (((!is_munmap && msg_idx->type == message_type_allocation) - || (is_munmap && msg_idx->type == message_type_mmap)) - && alloc_message_queue_[msg_idx->index].ptr == ptr) { - msg_idx->type = message_type_nil; - msg_queue_idx_--; - alloc_msg_queue_idx_--; - return; - } - } - - if (UNLIKELY(msg_queue_idx_ >= msg_queue_size_)) { - if (UNLIKELY(!buffer_realloc_message_queue(false))) { - return; - } - } - - message_t *msg_idx = &message_queue_[msg_queue_idx_++]; - msg_idx->type = is_munmap ? message_type_munmap : message_type_deletion; - msg_idx->ptr = ptr; - } + const size_t size_augment_; + const size_t memory_limit_; - inline bool empty() const { - return msg_queue_idx_ == 0; - } + size_t size_ = 0; + size_t idx_ = 0; + T *queue_ = nullptr; inline void reset() { - msg_queue_idx_ = 0; - alloc_msg_queue_idx_ = 0; - if (msg_queue_size_ > size_augment_ * 2) { - buffer_realloc_message_queue(true); - } - if (alloc_queue_size_ > size_augment_) { - buffer_realloc_alloc_queue(true); + idx_ = 0; + if (size_ > size_augment_) { + buffer_realloc(true); } } - inline size_t size() { - return msg_queue_idx_; - } - - inline void process(std::function callback) { - for (size_t i = 0; i < msg_queue_idx_; i++) { - message_t *message = &message_queue_[i]; - allocation_message_t *allocation_message = nullptr; - if (message->type == message_type_allocation || - message->type == message_type_mmap) { - allocation_message = &alloc_message_queue_[message->index]; + inline bool check_realloc() { + if (UNLIKELY(idx_ >= size_)) { + if (UNLIKELY(!buffer_realloc(false))) { + return false; } - callback(message, allocation_message); } - } - - // Statistic - static std::atomic g_queue_realloc_memory_1_counter; - static std::atomic g_queue_realloc_memory_2_counter; - static std::atomic g_queue_realloc_reason_1_counter; - static std::atomic g_queue_realloc_reason_2_counter; - static std::atomic g_queue_realloc_failure_counter; - static std::atomic g_queue_realloc_over_limit_counter; - static std::atomic g_queue_extra_stack_meta_allocated; - static std::atomic g_queue_extra_stack_meta_kept; - private: + return true; + } - bool buffer_realloc_message_queue(bool init) { - if (!init && msg_queue_size_ != 0) { - g_queue_realloc_reason_1_counter.fetch_add(1); - } + inline bool buffer_realloc(bool init) { - if (UNLIKELY(!init && g_queue_realloc_memory_1_counter.load() + - g_queue_realloc_memory_2_counter.load() >= MEMORY_OVER_LIMIT)) { - g_queue_realloc_over_limit_counter.fetch_add(1); + if (UNLIKELY(!init && + counter_->realloc_memory_counter_.load(std::memory_order_relaxed) >= + memory_limit_)) { + counter_->realloc_reach_limit_counter_.fetch_add(1, std::memory_order_relaxed); return false; } - size_t resize = msg_queue_size_; - if (init) { - resize = size_augment_ * 2; - } else { - resize += size_augment_ * 2; + if (!init && size_ != 0) { + counter_->realloc_counter_.fetch_add(1, std::memory_order_relaxed); } - void *buffer = realloc(message_queue_, sizeof(message_t) * resize); + size_t resize = init ? size_augment_ : (size_ + size_augment_); - if (buffer) { - message_queue_ = static_cast(buffer); + void *buffer = realloc(queue_, sizeof(T) * resize); - g_queue_realloc_memory_1_counter.fetch_sub(sizeof(message_t) * msg_queue_size_); - msg_queue_size_ = resize; - g_queue_realloc_memory_1_counter.fetch_add(sizeof(message_t) * msg_queue_size_); + if (LIKELY(buffer)) { + queue_ = static_cast(buffer); + + counter_->realloc_memory_counter_.fetch_sub(sizeof(T) * size_); + size_ = resize; + counter_->realloc_memory_counter_.fetch_add(sizeof(T) * size_); return true; } else { - g_queue_realloc_failure_counter.fetch_add(1, std::memory_order_relaxed); - + counter_->realloc_failure_counter_.fetch_add(1, std::memory_order_relaxed); return false; } } + }; - bool buffer_realloc_alloc_queue(bool init) { - if (!init && alloc_queue_size_ != 0) { - g_queue_realloc_reason_2_counter.fetch_add(1); - } - - if (UNLIKELY(!init && g_queue_realloc_memory_1_counter.load() + - g_queue_realloc_memory_2_counter.load() >= MEMORY_OVER_LIMIT)) { - g_queue_realloc_over_limit_counter.fetch_add(1); - return false; - } - - size_t resize = alloc_queue_size_; - if (init) { - resize = size_augment_; - } else { - resize += size_augment_; - } + class BufferQueue { - void *buffer = realloc(alloc_message_queue_, - sizeof(allocation_message_t) * resize); - if (buffer) { - alloc_message_queue_ = static_cast(buffer); + public: + BufferQueue(size_t size_augment) { + const size_t limit_size = (MEMORY_OVER_LIMIT / + (sizeof(message_t) * 2 + sizeof(allocation_message_t))); + + messages_ = new MessageAllocator( + size_augment * 2, + limit_size * 2 * sizeof(message_t), + &g_message_queue_counter_); + allocations_ = new MessageAllocator( + size_augment, + limit_size * sizeof(allocation_message_t), + &g_allocation_queue_counter_); + messages_->buffer_realloc(true); + allocations_->buffer_realloc(true); + }; - g_queue_realloc_memory_2_counter.fetch_sub( - sizeof(allocation_message_t) * alloc_queue_size_); - alloc_queue_size_ = resize; - g_queue_realloc_memory_2_counter.fetch_add( - sizeof(allocation_message_t) * alloc_queue_size_); + ~BufferQueue() { + delete messages_; + delete allocations_; + }; - return true; - } else { - g_queue_realloc_failure_counter.fetch_add(1, std::memory_order_relaxed); + inline allocation_message_t *enqueue_allocation_message(message_type type) { - return false; + if (UNLIKELY(!messages_->check_realloc()) || + UNLIKELY(!allocations_->check_realloc())) { + return nullptr; } - } - - size_t size_augment_ = 0; - size_t msg_queue_size_ = 0; - size_t msg_queue_idx_ = 0; - message_t *message_queue_ = nullptr; + CRITICAL_CHECK(type == message_type_mmap || type == message_type_reallocation || + type == message_type_allocation); - size_t alloc_queue_size_ = 0; - size_t alloc_msg_queue_idx_ = 0; - allocation_message_t *alloc_message_queue_ = nullptr; + message_t *msg_idx = &messages_->queue_[messages_->idx_++]; + msg_idx->type = type; + msg_idx->index = allocations_->idx_; - }; - -#else + allocation_message_t *buffer = &allocations_->queue_[allocations_->idx_++]; + *buffer = {}; + return buffer; + } - // Statistic - extern std::atomic g_queue_realloc_counter; - extern std::atomic g_queue_realloc_size_counter; - extern std::atomic g_queue_realloc_failure_counter; - extern std::atomic g_queue_realloc_over_limit_counter; - extern std::atomic g_queue_extra_stack_meta_allocated; - extern std::atomic g_queue_extra_stack_meta_kept; + inline bool enqueue_deletion_message(uintptr_t ptr, bool is_munmap) { - typedef Node message_node_t; + // Fast path. + if (messages_->idx_ > 0) { + message_t *msg_idx = &messages_->queue_[messages_->idx_ - 1]; + if (((!is_munmap && msg_idx->type == message_type_allocation) + || (is_munmap && msg_idx->type == message_type_mmap)) + && allocations_->queue_[msg_idx->index].ptr == ptr) { + msg_idx->type = message_type_nil; + messages_->idx_--; + allocations_->idx_--; + return true; + } + } - class BufferQueue { + if (UNLIKELY(!messages_->check_realloc())) { + return false; + } - public: - BufferQueue(FreeList * free_list) { - message_queue_ = new LockFreeQueue(free_list); - }; + message_t *msg_idx = &messages_->queue_[messages_->idx_++]; + msg_idx->type = is_munmap ? message_type_munmap : message_type_deletion; + msg_idx->ptr = ptr; - ~BufferQueue() { - delete message_queue_; + return true; } - inline void process(std::function *, Node *)> callback) { - - message_node_t * message_node = nullptr; + inline bool empty() const { + return messages_->idx_ == 0; + } - while (message_queue_->poll(message_node)) { + inline void reset() { + messages_->reset(); + allocations_->reset(); + } - CRITICAL_CHECK(message_node); + inline size_t size() { + return messages_->idx_; + } - Node * allocation_message_node = nullptr; - if (message_node->t_.type == message_type_allocation || - message_node->t_.type == message_type_mmap) { - allocation_message_node = - reinterpret_cast *>(message_node->t_.index); - CRITICAL_CHECK(message_node->t_.index); - CRITICAL_CHECK(allocation_message_node); + inline void process(std::function callback) { + for (size_t i = 0; i < messages_->idx_; i++) { + message_t *message = &messages_->queue_[i]; + allocation_message_t *allocation_message = nullptr; + if (message->type == message_type_allocation || + message->type == message_type_reallocation || + message->type == message_type_mmap) { + allocation_message = &allocations_->queue_[message->index]; } - - callback(message_node, allocation_message_node); + callback(message, allocation_message); } } - LockFreeQueue *message_queue_; + MessageAllocator *messages_; + MessageAllocator *allocations_; + + // Statistic + static std::atomic g_queue_extra_stack_meta_allocated; + static std::atomic g_queue_extra_stack_meta_kept; + static message_queue_counter_t g_message_queue_counter_; + static message_queue_counter_t g_allocation_queue_counter_; private: - }; -#endif + }; class BufferQueueContainer { public: @@ -327,17 +272,19 @@ namespace matrix { } BufferQueue *queue_; -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE != true + std::mutex mutex_; static std::atomic g_locker_collision_counter; + static std::atomic g_message_overflow_counter; + inline void lock() { - if (!mutex_.try_lock()) { + if (UNLIKELY(!mutex_.try_lock())) { g_locker_collision_counter.fetch_add(1, std::memory_order_relaxed); mutex_.lock(); } - if (!queue_) { + if (UNLIKELY(!queue_)) { queue_ = new BufferQueue(SIZE_AUGMENT); } } @@ -345,7 +292,6 @@ namespace matrix { inline void unlock() { mutex_.unlock(); } -#endif }; class BufferManagement { @@ -358,20 +304,11 @@ namespace matrix { std::vector containers_; -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true - FreeList *message_allocator_; - FreeList *alloc_message_allocator_; - FreeList *node_allocator_; -#endif - private: [[noreturn]] static void process_routine(BufferManagement *this_); -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true -#else BufferQueue *queue_swapped_ = nullptr; -#endif memory_meta_container *memory_meta_container_ = nullptr; diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHook.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHook.cpp index 45b220716..edae73331 100644 --- a/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHook.cpp +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHook.cpp @@ -60,18 +60,12 @@ static size_t m_tracing_alloc_size_max = 0; static size_t m_stacktrace_log_threshold; -#if USE_MEMORY_MESSAGE_QUEUE == true static BufferManagement m_memory_messages_containers_(&m_memory_meta_container); void memory_hook_init() { m_memory_messages_containers_.start_process(); } -#else -void memory_hook_init() { -} -#endif - void enable_stacktrace(bool enable) { is_stacktrace_enabled = enable; } @@ -90,7 +84,7 @@ static inline bool should_do_unwind(size_t byte_count) { && (m_tracing_alloc_size_max == 0 || byte_count <= m_tracing_alloc_size_max)); } -#if USE_FAKE_BACKTRACE_DATA == true +#if ENABLE_FAKE_BACKTRACE_DATA == true #define do_unwind(frames, max_frames, frame_size) fake_unwind(frames, max_frames, frame_size); #else #define do_unwind(frames, max_frames, frame_size) \ @@ -109,7 +103,7 @@ static inline void on_acquire_memory( void *caller, void *ptr, size_t byte_count, - bool is_mmap) { + message_type type) { if (UNLIKELY(!ptr)) { LOGE(TAG, "on_acquire_memory: invalid pointer"); @@ -118,43 +112,14 @@ static inline void on_acquire_memory( ON_MEMORY_ALLOCATED((uint64_t) ptr, byte_count); +#if USE_ALLOC_COUNTER != true allocate_counter.fetch_add(1, std::memory_order::memory_order_relaxed); +#endif -#if USE_MEMORY_MESSAGE_QUEUE == true BufferQueueContainer *container = m_memory_messages_containers_.containers_[memory_ptr_hash( (uintptr_t) ptr)]; { - #if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true - - auto message_node = m_memory_messages_containers_.message_allocator_->allocate(); - auto allocation_message_node = m_memory_messages_containers_.alloc_message_allocator_->allocate(); - - if (UNLIKELY(!message_node || !allocation_message_node)) { - m_memory_messages_containers_.message_allocator_->deallocate(message_node); - m_memory_messages_containers_.alloc_message_allocator_->deallocate(allocation_message_node); - return; - } - - message_node->t_.type = is_mmap ? message_type_mmap : message_type_allocation; - message_node->t_.index = reinterpret_cast(allocation_message_node); - - CRITICAL_CHECK(message_node->t_.index); - - allocation_message_node->t_.ptr = reinterpret_cast(ptr); - allocation_message_node->t_.size = byte_count; - allocation_message_node->t_.caller = reinterpret_cast(caller); - - if (LIKELY(is_stacktrace_enabled && should_do_unwind(byte_count))) { - size_t frame_size = 0; - do_unwind(allocation_message_node->t_.backtrace.frames, MEMHOOK_BACKTRACE_MAX_FRAMES, - frame_size); - allocation_message_node->t_.backtrace.frame_size = frame_size; - } - - container->queue_->message_queue_->offer(message_node); - - #else - memory_backtrace_t backtrace {0}; + memory_backtrace_t backtrace{0}; if (LIKELY(byte_count > 0 && is_stacktrace_enabled && should_do_unwind(byte_count))) { size_t frame_size = 0; do_unwind(backtrace.frames, MEMHOOK_BACKTRACE_MAX_FRAMES, @@ -164,8 +129,12 @@ static inline void on_acquire_memory( container->lock(); - auto message = container->queue_->enqueue_allocation_message(is_mmap); - if (LIKELY(message)) { + auto message = container->queue_->enqueue_allocation_message(type); + if (UNLIKELY(!message)) { + BufferQueueContainer::g_message_overflow_counter.fetch_add(1, + std::memory_order_relaxed); + CHECK_MESSAGE_OVERFLOW(!message); + } else { message->ptr = reinterpret_cast(ptr); message->size = byte_count; message->caller = reinterpret_cast(caller); @@ -175,38 +144,8 @@ static inline void on_acquire_memory( } container->unlock(); - #endif - } -#else - matrix::memory_backtrace_t backtrace; - uint64_t stack_hash = 0; - if (LIKELY(is_stacktrace_enabled && should_do_unwind(byte_count))) { - size_t frame_size = 0; - do_unwind(backtrace.frames, MEMHOOK_BACKTRACE_MAX_FRAMES, frame_size); - backtrace.frame_size = frame_size; - stack_hash = hash_frames(backtrace.frames, backtrace.frame_size); - } - m_memory_meta_container.insert( - ptr, - stack_hash, - [&](ptr_meta_t *ptr_meta, stack_meta_t *stack_meta) { - ptr_meta->ptr = ptr; - ptr_meta->size = byte_count; - ptr_meta->caller = caller; - ptr_meta->is_mmap = is_mmap; - - if (UNLIKELY(!stack_meta)) { - return; - } - - stack_meta->size += byte_count; - if (stack_meta->backtrace.frame_size == 0 && backtrace.frame_size != 0) { - stack_meta->backtrace = backtrace; - stack_meta->caller = caller; - } - }); -#endif + } } @@ -219,42 +158,32 @@ static inline void on_release_memory(void *ptr, bool is_munmap) { ON_MEMORY_RELEASED((uint64_t) ptr); +#if USE_ALLOC_COUNTER != true release_counter.fetch_add(1, std::memory_order::memory_order_relaxed); +#endif -#if USE_MEMORY_MESSAGE_QUEUE == true BufferQueueContainer *container = m_memory_messages_containers_.containers_[memory_ptr_hash( (uintptr_t) ptr)]; - #if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true - - auto message_node = m_memory_messages_containers_.message_allocator_->allocate(); - - if (UNLIKELY(!message_node)) return; + container->lock(); - message_node->t_.type = is_munmap ? message_type_munmap : message_type_deletion; - message_node->t_.ptr = reinterpret_cast(ptr); + bool ret = container->queue_->enqueue_deletion_message(reinterpret_cast(ptr), + is_munmap); - container->queue_->message_queue_->offer(message_node); + container->unlock(); - #else - if (!container->mutex_.try_lock()) { - BufferQueueContainer::g_locker_collision_counter.fetch_add(1, std::memory_order_relaxed); - container->mutex_.lock(); + if (UNLIKELY(!ret)) { + BufferQueueContainer::g_message_overflow_counter.fetch_add(1, std::memory_order_relaxed); + CHECK_MESSAGE_OVERFLOW(!ret); } - - if (!container->queue_) { - container->queue_ = new BufferQueue(SIZE_AUGMENT); - } - container->queue_->enqueue_deletion_message(reinterpret_cast(ptr), is_munmap); - container->mutex_.unlock(); - #endif -#else - m_memory_meta_container.erase(ptr); -#endif } void on_alloc_memory(void *caller, void *ptr, size_t byte_count) { - on_acquire_memory(caller, ptr, byte_count, false); + on_acquire_memory(caller, ptr, byte_count, message_type_allocation); +} + +void on_realloc_memory(void *caller, void *ptr, size_t byte_count) { + on_acquire_memory(caller, ptr, byte_count, message_type_reallocation); } void on_free_memory(void *ptr) { @@ -262,7 +191,7 @@ void on_free_memory(void *ptr) { } void on_mmap_memory(void *caller, void *ptr, size_t byte_count) { - on_acquire_memory(caller, ptr, byte_count, true); + on_acquire_memory(caller, ptr, byte_count, message_type_mmap); } void on_munmap_memory(void *ptr) { @@ -293,16 +222,14 @@ static inline size_t collect_metas(std::map &heap_caller_ meta->attr.is_mmap ? mmap_caller_metas : heap_caller_metas; auto &dest_stack_metas = meta->attr.is_mmap ? mmap_stack_metas : heap_stack_metas; - void * caller; -#if USE_STACK_HASH_NO_COLLISION == true + void *caller; + if (stack_meta) { caller = reinterpret_cast(stack_meta->caller); } else { caller = reinterpret_cast(meta->caller); } -#else - caller = reinterpret_cast(meta->caller); -#endif + if (caller) { caller_meta_t &caller_meta = dest_caller_metes[caller]; caller_meta.pointers.insert(ptr); @@ -310,11 +237,9 @@ static inline size_t collect_metas(std::map &heap_caller_ } if (stack_meta) { -#if USE_STACK_HASH_NO_COLLISION == true - auto &dest_stack_meta = dest_stack_metas[(uint64_t)stack_meta]; -#else - auto &dest_stack_meta = dest_stack_metas[meta->stack_hash]; -#endif + + auto &dest_stack_meta = dest_stack_metas[(uint64_t) stack_meta]; + dest_stack_meta.backtrace = stack_meta->backtrace; // 没错, 这里的确使用 ptr_meta 的 size, 因为是在遍历 ptr_meta, 因此原来 stack_meta 的 size 仅起引用计数作用 dest_stack_meta.size += meta->size; @@ -356,7 +281,7 @@ static inline void dump_callers(FILE *log_file, Dl_info dl_info; dladdr(caller, &dl_info); - const char* fname = ""; + const char *fname = ""; if (dl_info.dli_fname) { fname = dl_info.dli_fname; } @@ -423,49 +348,44 @@ static inline void dump_callers(FILE *log_file, LOGD(TAG, "\n---------------------------------------------------"); flogger0(log_file, "\n---------------------------------------------------\n"); LOGD(TAG, "| Caller total size = %zu bytes", caller_total_size); - flogger0(log_file, "| Caller total size = %zu bytes, ptr total counts = %zu.\n", caller_total_size, + flogger0(log_file, "| Caller total size = %zu bytes, ptr total counts = %zu.\n", + caller_total_size, ptr_counter); flogger0(log_file, "| Allocation times = %zu, release times = %zu.\n", allocate_counter.load(std::memory_order_relaxed), release_counter.load(std::memory_order_relaxed)); -#if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true - flogger0(log_file, - "| Container realloc times = %zu, queue realloc = %zu.\n", - buffer_source_memory::g_realloc_counter.load(std::memory_order_relaxed), - g_queue_realloc_counter.load(std::memory_order_relaxed)); - flogger0(log_file, - "| Container realloc = %zu bytes, queue realloc = %zu bytes.\n", - buffer_source_memory::g_realloc_memory_counter.load(std::memory_order_relaxed), - g_queue_realloc_size_counter.load(std::memory_order_relaxed)); - flogger0(log_file, "| Realloc failure = %zu, memory over limit failure = %zu.\n", - g_queue_realloc_failure_counter.load(std::memory_order_relaxed), - g_queue_realloc_over_limit_counter.load(std::memory_order_relaxed)); - flogger0(log_file, "| Hash extra allocated = %zu, kept = %zu, kept size = %zu byte.\n", - g_queue_extra_stack_meta_allocated.load(std::memory_order_relaxed), - g_queue_extra_stack_meta_kept.load(std::memory_order_relaxed), - g_queue_extra_stack_meta_kept.load(std::memory_order_relaxed) * sizeof(stack_meta_t)); -#else flogger0(log_file, "| Container realloc times = %zu, queue realloc reason-1 = %zu, reason-2 = %zu.\n", buffer_source_memory::g_realloc_counter.load(std::memory_order_relaxed), - BufferQueue::g_queue_realloc_reason_1_counter.load(std::memory_order_relaxed), - BufferQueue::g_queue_realloc_reason_2_counter.load(std::memory_order_relaxed)); + BufferQueue::g_message_queue_counter_.realloc_counter_.load(std::memory_order_relaxed), + BufferQueue::g_allocation_queue_counter_.realloc_counter_.load( + std::memory_order_relaxed)); flogger0(log_file, "| Container realloc = %zu bytes, queue realloc reason-1 = %zu bytes, reason-2 = %zu bytes.\n", buffer_source_memory::g_realloc_memory_counter.load(std::memory_order_relaxed), - BufferQueue::g_queue_realloc_memory_1_counter.load(std::memory_order_relaxed), - BufferQueue::g_queue_realloc_memory_2_counter.load(std::memory_order_relaxed)); - flogger0(log_file, "| Queue lock collision = %zu.\n", - BufferQueueContainer::g_locker_collision_counter.load(std::memory_order_relaxed)); - flogger0(log_file, "| Realloc failure = %zu, memory over limit failure = %zu.\n", - BufferQueue::g_queue_realloc_failure_counter.load(std::memory_order_relaxed), - BufferQueue::g_queue_realloc_over_limit_counter.load(std::memory_order_relaxed)); - flogger0(log_file, "| Hash extra allocated = %zu, kept = %zu, kept size = %zu byte.\n", + BufferQueue::g_message_queue_counter_.realloc_memory_counter_.load( + std::memory_order_relaxed), + BufferQueue::g_allocation_queue_counter_.realloc_memory_counter_.load( + std::memory_order_relaxed)); + flogger0(log_file, "| Queue lock collisions = %zu, messages overflow = %zu.\n", + BufferQueueContainer::g_locker_collision_counter.load(std::memory_order_relaxed), + BufferQueueContainer::g_message_overflow_counter.load(std::memory_order_relaxed)); + flogger0(log_file, "| Realloc failures = %zu, memory over limit failures = %zu.\n", + BufferQueue::g_message_queue_counter_.realloc_failure_counter_.load( + std::memory_order_relaxed) + + BufferQueue::g_allocation_queue_counter_.realloc_failure_counter_.load( + std::memory_order_relaxed), + BufferQueue::g_message_queue_counter_.realloc_reach_limit_counter_.load( + std::memory_order_relaxed) + + BufferQueue::g_allocation_queue_counter_.realloc_reach_limit_counter_.load( + std::memory_order_relaxed)); + flogger0(log_file, "| Hash extra allocated = %zu, kept = %zu, kept size = %zu bytes.\n", BufferQueue::g_queue_extra_stack_meta_allocated.load(std::memory_order_relaxed), BufferQueue::g_queue_extra_stack_meta_kept.load(std::memory_order_relaxed), - BufferQueue::g_queue_extra_stack_meta_kept.load(std::memory_order_relaxed) * sizeof(stack_meta_t)); -#endif + BufferQueue::g_queue_extra_stack_meta_kept.load(std::memory_order_relaxed) * + sizeof(stack_meta_t)); + LOGD(TAG, "---------------------------------------------------\n"); flogger0(log_file, "---------------------------------------------------\n\n"); } @@ -495,7 +415,7 @@ static inline void dump_stacks(FILE *log_file, for (auto &stack_meta_it : stack_metas) { auto size = stack_meta_it.second.size; auto backtrace = stack_meta_it.second.backtrace; - void* caller = reinterpret_cast(stack_meta_it.second.caller); + void *caller = reinterpret_cast(stack_meta_it.second.caller); std::string caller_so_name; std::stringstream full_stack_builder; diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHook.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHook.h index a07e0ce9e..634784c42 100644 --- a/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHook.h +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHook.h @@ -29,6 +29,8 @@ extern "C" void fake_free(void * ptr); void on_alloc_memory(void *caller, void *ptr, size_t byte_count); +void on_realloc_memory(void *caller, void *ptr, size_t byte_count); + void on_free_memory(void *ptr); void on_mmap_memory(void *caller, void *ptr, size_t byte_count); diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHookFunctions.cpp b/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHookFunctions.cpp index 407ea5c14..6755b58e3 100644 --- a/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHookFunctions.cpp +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHookFunctions.cpp @@ -66,12 +66,10 @@ DEFINE_HOOK_FUN(void *, realloc, void *__ptr, size_t __byte_count) { return p; } - // whatever has been moved or not, record anyway, because using realloc to shrink an allocation is allowed. - // wtf whatever. if p == __ptr , just override meta - if (p != __ptr) { - on_free_memory(__ptr); - } LOGI(TAG, "+ realloc2 %p", p); + + // whatever has been moved or not, record anyway, because using realloc to shrink an allocation is allowed. + on_free_memory(__ptr); on_alloc_memory(caller, p, __byte_count); return p; diff --git a/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHookMetas.h b/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHookMetas.h index ea7d23520..954abebe7 100644 --- a/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHookMetas.h +++ b/matrix/matrix-android/matrix-hooks/src/main/cpp/memory/MemoryHookMetas.h @@ -36,7 +36,7 @@ struct __attribute__((__packed__)) ptr_meta_t { void *ptr; size_t size; uint64_t stack_hash; -#if USE_STACK_HASH_NO_COLLISION == true + union { uintptr_t caller; uintptr_t stack_idx; @@ -46,12 +46,6 @@ struct __attribute__((__packed__)) ptr_meta_t { unsigned char is_stack_idx : 1; unsigned char is_mmap : 1; } attr; -#else - uintptr_t caller; - struct { - unsigned char is_mmap : 1; - } attr; -#endif }; struct caller_meta_t { @@ -66,197 +60,12 @@ struct __attribute__((__packed__)) stack_meta_t { size_t size; uintptr_t caller; matrix::memory_backtrace_t backtrace; -#if USE_STACK_HASH_NO_COLLISION == true void *ext; -#endif }; typedef splay_map memory_map_t; typedef splay_map stack_map_t; -#if USE_MEMORY_MESSAGE_QUEUE == false - -class memory_meta_container { - - typedef struct { - memory_map_t container = memory_map_t(64); - std::mutex mutex; - } ptr_meta_container_wrapper_t; - - typedef struct { -#if USE_SPLAY_MAP_SAVE_STACK == true - stack_map_t container = stack_map_t(8); -#else - std::map container; -#endif - std::mutex mutex; - } stack_container_wrapper_t; - -#define TARGET_PTR_CONTAINER_LOCKED(target, key) \ - ptr_meta_container_wrapper_t * target = ptr_meta_containers.data()[ptr_meta_hash((uintptr_t) key)]; \ - std::lock_guard target_lock(target->mutex) - -#define TARGET_STACK_CONTAINER_LOCKED(target, key) \ - stack_container_wrapper_t *target = stack_meta_containers.data()[stack_meta_hash(key)]; \ - std::lock_guard stack_lock(target->mutex) - -public: - - memory_meta_container() { - size_t cap = ptr_meta_capacity(); - ptr_meta_containers.reserve(cap); - for (int i = 0; i < cap; ++i) { - ptr_meta_containers.emplace_back(new ptr_meta_container_wrapper_t); - } - - cap = stack_meta_capacity(); - stack_meta_containers.reserve(cap); - for (int i = 0; i < cap; ++i) { - stack_meta_containers.emplace_back(new stack_container_wrapper_t); - } - } - - inline void insert(const void *__ptr, - uint64_t __stack_hash, - std::function __callback) { - TARGET_PTR_CONTAINER_LOCKED(ptr_meta_container, __ptr); - auto ptr_meta = ptr_meta_container->container.insert(__ptr, { 0 }); - - if (UNLIKELY(ptr_meta == nullptr)) { - return; - } - - ptr_meta->stack_hash = __stack_hash; - - if (__stack_hash) { - stack_meta_t *stack_meta; - TARGET_STACK_CONTAINER_LOCKED(stack_meta_container, __stack_hash); -#if USE_SPLAY_MAP_SAVE_STACK == true - if (LIKELY(stack_meta_container->container.exist(__stack_hash))) { - stack_meta = &stack_meta_container->container.find(); - } else { - stack_meta = stack_meta_container->container.insert(__stack_hash, {0}); - } -#else - auto it = stack_meta_container->container.find(__stack_hash); - if (LIKELY(it != stack_meta_container->container.end())) { - stack_meta = &it->second; - } else { - stack_meta = &stack_meta_container->container[__stack_hash]; - } -#endif - __callback(ptr_meta, stack_meta); - } else { - __callback(ptr_meta, nullptr); - } - } - - template - inline void get(const void *__k, _Callable __callable) { - TARGET_PTR_CONTAINER_LOCKED(target, __k); - if (target->container.exist(__k)) { - auto &meta = target->container.find(); - __callable(meta); - } - } - - inline bool erase(const void *__k) { - TARGET_PTR_CONTAINER_LOCKED(ptr_meta_container, __k); - - auto removed_ptr_meta = ptr_meta_container->container.remove(__k); - - if (UNLIKELY(!removed_ptr_meta)) { // not contains - return false; - } - - auto ptr_meta = *removed_ptr_meta; - - if (LIKELY(ptr_meta.stack_hash)) { - TARGET_STACK_CONTAINER_LOCKED(stack_meta_container, ptr_meta.stack_hash); -#if USE_SPLAY_MAP_SAVE_STACK == true - if (LIKELY(stack_meta_container->container.exist(ptr_meta.stack_hash))) { - auto &stack_meta = stack_meta_container->container.find(); - if (stack_meta.size > ptr_meta.size) { // 减去同堆栈的 size - stack_meta.size -= ptr_meta.size; - } else { // 删除 size 为 0 的堆栈 - stack_meta_container->container.remove(ptr_meta.stack_hash); - } - } -#else - auto it = stack_meta_container->container.find(ptr_meta.stack_hash); - if (LIKELY(it != stack_meta_container->container.end())) { - auto &stack_meta = it->second; - if (stack_meta.size > ptr_meta.size) { // 减去同堆栈的 size - stack_meta.size -= ptr_meta.size; - } else { // 删除 size 为 0 的堆栈 - stack_meta_container->container.erase(it); - } - } -#endif - } - - return true; - } - - bool contains(const void *__k) { - TARGET_PTR_CONTAINER_LOCKED(ptr_meta_container, __k); - return ptr_meta_container->container.exist(__k); - } - - void for_each(std::function __callback) { - for (const auto cw : ptr_meta_containers) { - std::lock_guard container_lock(cw->mutex); - cw->container.enumerate([&](const void * ptr, ptr_meta_t &ptr_meta) { - if (ptr_meta.stack_hash) { - TARGET_STACK_CONTAINER_LOCKED(stack_meta_container, ptr_meta.stack_hash); - stack_meta_t *stack_meta = nullptr; -#if USE_SPLAY_MAP_SAVE_STACK == true - if (stack_meta_container->container.exist(ptr_meta.stack_hash)) { - stack_meta = &stack_meta_container->container.find(); - } -#else - auto it = stack_meta_container->container.find(ptr_meta.stack_hash); - if (it != stack_meta_container->container.end()) { - stack_meta = &it->second; - } -#endif - __callback(ptr, &ptr_meta, stack_meta); // within lock scope - } else { - __callback(ptr, &ptr_meta, nullptr); - } - }); - } - } - -private: - - static inline size_t ptr_meta_capacity() { - return MAX_PTR_META_SLOT; - } - - static inline size_t ptr_meta_hash(uintptr_t __key) { - return static_cast((__key ^ (__key >> 16)) & PTR_META_MASK); - } - - static inline size_t stack_meta_capacity() { - return MAX_STACK_META_SLOT; - } - - static inline size_t stack_meta_hash(uint64_t __key) { - return ((__key ^ (__key >> 16)) & STACK_META_MASK); - } - - std::vector ptr_meta_containers; - std::vector stack_meta_containers; - - static const unsigned int MAX_PTR_META_SLOT = 1 << 10; - static const unsigned int PTR_META_MASK = MAX_PTR_META_SLOT - 1; - static const unsigned int MAX_STACK_META_SLOT = 1 << 9; - static const unsigned int STACK_META_MASK = MAX_STACK_META_SLOT - 1; -}; - -#else - class memory_meta_container { typedef struct { @@ -265,11 +74,7 @@ class memory_meta_container { } ptr_meta_container_wrapper_t; typedef struct { -#if USE_SPLAY_MAP_SAVE_STACK == true stack_map_t container = stack_map_t(STACK_SPLAY_MAP_CAPACITY); -#else - std::map container; -#endif std::mutex mutex; } stack_container_wrapper_t; @@ -319,12 +124,10 @@ class memory_meta_container { inline void insert( const void *__ptr, uint64_t __stack_hash, -#if USE_SPLAY_MAP_SAVE_STACK == true matrix::allocation_message_t *allocation_message, -#endif std::function __callback) { - TARGET_PTR_CONTAINER_LOCKED(ptr_meta_container, __ptr); + TARGET_PTR_CONTAINER_LOCKED(ptr_meta_container, __ptr); auto ptr_meta = ptr_meta_container->container.insert(__ptr, {0}); if (UNLIKELY(ptr_meta == nullptr)) { // Commonly no memory @@ -335,11 +138,10 @@ class memory_meta_container { if (__stack_hash) { stack_meta_t *stack_meta; + TARGET_STACK_CONTAINER_LOCKED(stack_meta_container, __stack_hash); -#if USE_SPLAY_MAP_SAVE_STACK == true if (LIKELY(stack_meta_container->container.exist(__stack_hash))) { stack_meta = &stack_meta_container->container.find(); - #if USE_STACK_HASH_NO_COLLISION == true uint32_t last_ptr = stack_meta_container->container.root_ptr(); bool same = false; @@ -366,20 +168,16 @@ class memory_meta_container { target = stack_meta; // Reuse first stack_meta. is_top = 1; } else { - target->ext = malloc(sizeof(stack_meta_t)); - *static_cast(target->ext) = {0}; + target->ext = calloc(1, sizeof(stack_meta_t)); target = static_cast(target->ext); - #if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true - // Statistic - matrix::g_queue_extra_stack_meta_allocated.fetch_add(1, std::memory_order_relaxed); - matrix::g_queue_extra_stack_meta_kept.fetch_add(1, std::memory_order_relaxed); - #else is_top = 0; + // Statistic - matrix::BufferQueue::g_queue_extra_stack_meta_allocated.fetch_add(1, std::memory_order_relaxed); - matrix::BufferQueue::g_queue_extra_stack_meta_kept.fetch_add(1, std::memory_order_relaxed); - #endif + matrix::BufferQueue::g_queue_extra_stack_meta_allocated.fetch_add( + 1, std::memory_order_relaxed); + matrix::BufferQueue::g_queue_extra_stack_meta_kept.fetch_add( + 1, std::memory_order_relaxed); } } @@ -390,25 +188,13 @@ class memory_meta_container { ptr_meta->ext_stack_ptr = reinterpret_cast(target); } stack_meta = target; - - #endif } else { stack_meta = stack_meta_container->container.insert(__stack_hash, {0}); - #if USE_STACK_HASH_NO_COLLISION == true if (stack_meta) { ptr_meta->stack_idx = stack_meta_container->container.root_ptr(); ptr_meta->attr.is_stack_idx = 1; } - #endif - } -#else - auto it = stack_meta_container->container.find(__stack_hash); - if (LIKELY(it != stack_meta_container->container.end())) { - stack_meta = &it->second; - } else { - stack_meta = &stack_meta_container->container[__stack_hash]; } -#endif CRITICAL_CHECK(stack_meta); __callback(ptr_meta, stack_meta); @@ -419,6 +205,7 @@ class memory_meta_container { template inline void get(const void *__k, _Callable __callable) { + TARGET_PTR_CONTAINER_LOCKED(target, __k); if (target->container.exist(__k)) { auto &meta = target->container.find(); @@ -427,8 +214,8 @@ class memory_meta_container { } inline bool erase(const void *__k) { - TARGET_PTR_CONTAINER_LOCKED(ptr_meta_container, __k); + TARGET_PTR_CONTAINER_LOCKED(ptr_meta_container, __k); auto removed_ptr_meta = ptr_meta_container->container.remove(__k); if (UNLIKELY(!removed_ptr_meta)) { // not contains @@ -438,10 +225,9 @@ class memory_meta_container { auto ptr_meta = *removed_ptr_meta; if (LIKELY(ptr_meta.stack_hash)) { + TARGET_STACK_CONTAINER_LOCKED(stack_meta_container, ptr_meta.stack_hash); -#if USE_SPLAY_MAP_SAVE_STACK == true if (LIKELY(stack_meta_container->container.exist(ptr_meta.stack_hash))) { - #if USE_STACK_HASH_NO_COLLISION == true auto &top_stack_meta = stack_meta_container->container.find(); auto top_stack_idx = stack_meta_container->container.root_ptr(); stack_meta_t *stack_meta; @@ -485,41 +271,20 @@ class memory_meta_container { if (ext) { prev->ext = ext->ext; free(ext); - #if USE_MEMORY_MESSAGE_QUEUE_LOCK_FREE == true - matrix::g_queue_extra_stack_meta_kept.fetch_sub(1, std::memory_order_relaxed); - #else + // Statistic matrix::BufferQueue::g_queue_extra_stack_meta_kept.fetch_sub(1, std::memory_order_relaxed); - #endif } } } - #else - auto &stack_meta = stack_meta_container->container.find(); - if (stack_meta.size > ptr_meta.size) { // 减去同堆栈的 size - stack_meta.size -= ptr_meta.size; - } else { // 删除 size 为 0 的堆栈 - stack_meta_container->container.remove(ptr_meta.stack_hash); - } - #endif - } -#else - auto it = stack_meta_container->container.find(ptr_meta.stack_hash); - if (LIKELY(it != stack_meta_container->container.end())) { - auto &stack_meta = it->second; - if (stack_meta.size > ptr_meta.size) { // 减去同堆栈的 size - stack_meta.size -= ptr_meta.size; - } else { // 删除 size 为 0 的堆栈 - stack_meta_container->container.erase(it); - } } -#endif } return true; } bool contains(const void *__k) { + TARGET_PTR_CONTAINER_LOCKED(ptr_meta_container, __k); return ptr_meta_container->container.exist(__k); } @@ -529,26 +294,16 @@ class memory_meta_container { std::lock_guard container_lock(cw->mutex); cw->container.enumerate([&](const void *ptr, ptr_meta_t &ptr_meta) { if (ptr_meta.stack_hash) { + TARGET_STACK_CONTAINER_LOCKED(stack_meta_container, ptr_meta.stack_hash); stack_meta_t *stack_meta = nullptr; -#if USE_SPLAY_MAP_SAVE_STACK == true if (LIKELY(stack_meta_container->container.exist(ptr_meta.stack_hash))) { - #if USE_STACK_HASH_NO_COLLISION == true if (ptr_meta.attr.is_stack_idx) { stack_meta = &stack_meta_container->container.get(ptr_meta.stack_idx); } else { stack_meta = reinterpret_cast(ptr_meta.ext_stack_ptr); } - #else - stack_meta = &stack_meta_container->container.find(); - #endif } -#else - auto it = stack_meta_container->container.find(ptr_meta.stack_hash); - if (it != stack_meta_container->container.end()) { - stack_meta = &it->second; - } -#endif __callback(ptr, &ptr_meta, stack_meta); // within lock scope } else { __callback(ptr, &ptr_meta, nullptr); @@ -578,20 +333,13 @@ class memory_meta_container { std::vector ptr_meta_containers; std::vector stack_meta_containers; -#if USE_MEMORY_MESSAGE_QUEUE == true static const unsigned int MAX_PTR_META_SLOT = 1 << 0; static const unsigned int MAX_STACK_META_SLOT = 1 << 0; -#else - static const unsigned int MAX_PTR_META_SLOT = 1 << 10; - static const unsigned int MAX_STACK_META_SLOT = 1 << 9; -#endif static const unsigned int PTR_META_MASK = MAX_PTR_META_SLOT - 1; static const unsigned int STACK_META_MASK = MAX_STACK_META_SLOT - 1; }; -#endif - #undef TAG #endif //LIBMATRIX_HOOK_MEMORYHOOKMETAS_H diff --git a/matrix/matrix-android/matrix-hooks/src/main/java/com/tencent/matrix/hook/memory/MemoryHook.java b/matrix/matrix-android/matrix-hooks/src/main/java/com/tencent/matrix/hook/memory/MemoryHook.java index 64b633339..52183aa80 100644 --- a/matrix/matrix-android/matrix-hooks/src/main/java/com/tencent/matrix/hook/memory/MemoryHook.java +++ b/matrix/matrix-android/matrix-hooks/src/main/java/com/tencent/matrix/hook/memory/MemoryHook.java @@ -23,6 +23,7 @@ import com.tencent.matrix.hook.AbsHook; import com.tencent.matrix.hook.HookManager; +import com.tencent.matrix.memguard.MemGuard; import com.tencent.matrix.util.MatrixLog; import java.util.HashSet; @@ -126,6 +127,11 @@ protected String getNativeLibraryName() { @Override public boolean onConfigure() { + if (MemGuard.isInstalled()) { + MatrixLog.w(TAG, "MemGuard has been installed, skip MemoryHook install logic."); + return false; + } + if (mMinTraceSize < 0 || (mMaxTraceSize != 0 && mMaxTraceSize < mMinTraceSize)) { throw new IllegalArgumentException("sizes should not be negative and maxSize should be " + "0 or greater than minSize: min = " + mMinTraceSize + ", max = " + mMaxTraceSize); diff --git a/matrix/matrix-android/matrix-hooks/src/main/java/com/tencent/matrix/memguard/MemGuard.java b/matrix/matrix-android/matrix-hooks/src/main/java/com/tencent/matrix/memguard/MemGuard.java new file mode 100644 index 000000000..6d8079421 --- /dev/null +++ b/matrix/matrix-android/matrix-hooks/src/main/java/com/tencent/matrix/memguard/MemGuard.java @@ -0,0 +1,464 @@ +package com.tencent.matrix.memguard; + +import android.app.Activity; +import android.app.ActivityManager; +import android.content.Context; +import android.os.Process; +import android.text.TextUtils; + +import androidx.annotation.Keep; +import androidx.annotation.NonNull; +import androidx.annotation.Nullable; + +import com.tencent.matrix.hook.AbsHook; +import com.tencent.matrix.hook.memory.MemoryHook; +import com.tencent.matrix.util.MatrixLog; + +import java.io.BufferedReader; +import java.io.File; +import java.io.FileReader; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Objects; + +public final class MemGuard { + private static final String TAG = "MemGuard"; + + private static final String NATIVE_LIB_NAME = "matrix-memguard"; + private static final String ISSUE_CALLBACK_THREAD_NAME = "MemGuard.IssueCB"; + private static final long ISSUE_CALLBACK_TIMEOUT_MS = 5000; + + private static final boolean[] sInstalled = {false}; + + public interface NativeLibLoader { + void loadLibrary(@NonNull String libraryName); + } + + public interface IssueCallback { + void onIssueDumpped(@NonNull String dumpFile) throws Throwable; + } + + private static IssueCallback sIssueCallback = new IssueCallback() { + @Override + public void onIssueDumpped(@NonNull String dumpFile) throws Throwable { + final File fDumpFile = new File(dumpFile); + if (!fDumpFile.exists()) { + MatrixLog.e(TAG, "Dump file %s does not exist, dump failure ?", dumpFile); + return; + } + BufferedReader br = null; + try { + br = new BufferedReader(new FileReader(fDumpFile)); + String line = null; + while ((line = br.readLine()) != null) { + MatrixLog.w(TAG, "[DumpedIssue] >> %s", line); + } + } finally { + if (br != null) { + try { + br.close(); + } catch (Throwable ignored) { + // Ignored. + } + } + } + } + }; + + public static boolean install(@NonNull Options opts, @Nullable IssueCallback issueCallback) { + return install(opts, null, issueCallback); + } + + public static boolean install(@NonNull Options opts, + @Nullable NativeLibLoader soLoader, + @Nullable IssueCallback issueCallback) { + Objects.requireNonNull(opts); + + synchronized (sInstalled) { + if (sInstalled[0]) { + MatrixLog.w(TAG, "Already installed."); + return true; + } + + if (MemoryHook.INSTANCE.getStatus() == AbsHook.Status.COMMIT_SUCCESS) { + MatrixLog.w(TAG, "MemoryHook has been committed, skip MemGuard install logic."); + return false; + } + + boolean success = false; + try { + if (soLoader != null) { + soLoader.loadLibrary(NATIVE_LIB_NAME); + } else { + System.loadLibrary(NATIVE_LIB_NAME); + } + + if (issueCallback != null) { + sIssueCallback = issueCallback; + } + + success = nativeInstall(opts); + } catch (Throwable thr) { + MatrixLog.printErrStackTrace(TAG, thr, "Install MemGuard failed."); + success = false; + } + if (success) { + MatrixLog.i(TAG, "Install MemGuard successfully with " + opts); + } else { + MatrixLog.e(TAG, "Install MemGuard failed with " + opts); + } + sInstalled[0] = success; + return success; + } + } + + public static boolean isInstalled() { + synchronized (sInstalled) { + return sInstalled[0]; + } + } + + @Nullable + public static File getLastIssueDumpFileIfExists() { + final String issueDumpFilePath = nativeGetIssueDumpFilePath(); + if (TextUtils.isEmpty(issueDumpFilePath)) { + return null; + } + final File result = new File(issueDumpFilePath); + return result.exists() ? result : null; + } + + private static native boolean nativeInstall(@NonNull Options opts); + + @Nullable + private static native String nativeGetIssueDumpFilePath(); + + @Keep + private static void c2jNotifyOnIssueDumped(final String dumpFile) { + final Thread cbThread = new Thread(new Runnable() { + @Override + public void run() { + try { + sIssueCallback.onIssueDumpped(dumpFile); + } catch (Throwable thr) { + MatrixLog.printErrStackTrace(TAG, thr, "Exception was thrown when onIssueDumpped was called."); + } + } + }, ISSUE_CALLBACK_THREAD_NAME); + final long st = System.currentTimeMillis(); + cbThread.start(); + try { + cbThread.join(ISSUE_CALLBACK_TIMEOUT_MS); + } catch (InterruptedException e) { + MatrixLog.w(TAG, "Issue callback was interrupted."); + } + final long cost = System.currentTimeMillis() - st; + if (cost > ISSUE_CALLBACK_TIMEOUT_MS) { + MatrixLog.w(TAG, "Timeout when call issue callback."); + } + } + + public static final class Options { + public static final int DEFAULT_MAX_ALLOCATION_SIZE = 8 * 1024; // 8K + public static final int DEFAULT_MAX_DETECTABLE_ALLOCATION_COUNT = 4096; + public static final int DEFAULT_MAX_SKIPPED_ALLOCATION_COUNT = 5; + public static final int DEFAULT_PERCENTAGE_OF_LEFT_SIDE_GUARD = 30; + public static final boolean DEFAULT_PERFECT_RIGHT_SIDE_GUARD = false; + public static final boolean DEFAULT_IGNORE_OVERLAPPED_READING = false; + public static final String DEFAULT_TARGET_SO_PATTERN = ".*/lib.*\\.so$"; + + /** + * Max allocation size MemGuard can detect its under/overflow issues. + */ + @Keep + public int maxAllocationSize; + + /** + * Max allocation count MemGuard can detect its under/overflow issues. + */ + @Keep + public int maxDetectableAllocationCount; + + /** + * Max skipped allocation count between two guarded allocations. + * For example, if 5 was set to this option, MemGuard will generate a random number 'k' in range [0,5] and + * the first k-th allocations will be ignored. + */ + @Keep + public int maxSkippedAllocationCount; + + /** + * Probability of putting guard page on the left side of specific pointer. + * For example, if 30 was set to this option, the probability of a pointer being guarded on the left side + * will be 30%, and the probability of a pointer being guarded on the right side will be 70%. + */ + @Keep + public int percentageOfLeftSideGuard; + + /** + * Whether MemGuard should return a pointer with guard page on right side without gaps. If true was set to + * this option, overflow issue will be easier to be detected but the returned pointer may not be aligned + * properly. Sometimes these not aligned pointers can crash your app. + */ + @Keep + public boolean perfectRightSideGuard; + + /** + * Whether MemGuard should regard overlapped reading as an issue. + */ + @Keep + public boolean ignoreOverlappedReading; + + /** + * Path to write dump file when memory issue was detected. Leave it null or empty will omit dumping issue + * info into file. + */ + @Keep + public String issueDumpFilePath; + + /** + * Patterns described by RegEx of target libs that we want to detect any memory issues. + */ + @Keep + public String[] targetSOPatterns; + + /** + * Patterns described by RegEx of target libs that we want to skip for detecting any memory issues. + */ + @Keep + public String[] ignoredSOPatterns; + + @Override + public String toString() { + return "Options{" + + "maxAllocationSize=" + maxAllocationSize + + ", maxDetectableAllocationCount=" + maxDetectableAllocationCount + + ", maxSkippedAllocationCount=" + maxSkippedAllocationCount + + ", percentageOfLeftSideGuard=" + percentageOfLeftSideGuard + + ", perfectRightSideGuard=" + perfectRightSideGuard + + ", ignoreOverlappedReading=" + ignoreOverlappedReading + + ", issueDumpFilePath=" + issueDumpFilePath + + ", targetSOPatterns=" + Arrays.toString(targetSOPatterns) + + ", ignoredSOPatterns=" + Arrays.toString(ignoredSOPatterns) + + '}'; + } + + private Options() { + // Do nothing. + } + + public static class Builder { + private Context mContext; + private int mMaxAllocationSize; + private int mMaxDetectableAllocationCount; + private int mMaxSkippedAllocationCount; + private int mPercentageOfLeftSideGuard; + private boolean mPerfectRightSideGuard; + private boolean mIgnoreOverlappedReading; + private String mIssueDumpFileDir; + private final List mTargetSOPatterns; + private final List mIgnoredSOPatterns; + + public Builder(Context context) { + mContext = context; + if (mContext instanceof Activity) { + mContext = mContext.getApplicationContext(); + } + mMaxAllocationSize = DEFAULT_MAX_ALLOCATION_SIZE; + mMaxDetectableAllocationCount = DEFAULT_MAX_DETECTABLE_ALLOCATION_COUNT; + mMaxSkippedAllocationCount = DEFAULT_MAX_SKIPPED_ALLOCATION_COUNT; + mPercentageOfLeftSideGuard = DEFAULT_PERCENTAGE_OF_LEFT_SIDE_GUARD; + mPerfectRightSideGuard = DEFAULT_PERFECT_RIGHT_SIDE_GUARD; + mIgnoreOverlappedReading = DEFAULT_IGNORE_OVERLAPPED_READING; + mIssueDumpFileDir = getDefaultIssueDumpDir(context); + mTargetSOPatterns = new ArrayList<>(); + mIgnoredSOPatterns = new ArrayList<>(); + } + + /** + * @see Options#maxAllocationSize + */ + public int getMaxAllocationSize() { + return mMaxAllocationSize; + } + + /** + * @see Options#maxAllocationSize + */ + public @NonNull Builder setMaxDetectableSize(int value) { + mMaxAllocationSize = value; + return this; + } + + /** + * @see Options#maxDetectableAllocationCount + */ + public int getMaxDetectableAllocationCount() { + return mMaxDetectableAllocationCount; + } + + /** + * @see Options#maxDetectableAllocationCount + */ + public @NonNull Builder setMaxDetectableAllocationCount(int value) { + mMaxDetectableAllocationCount = value; + return this; + } + + /** + * @see Options#maxSkippedAllocationCount + */ + public int getMaxSkippedAllocationCount() { + return mMaxSkippedAllocationCount; + } + + /** + * @see Options#maxSkippedAllocationCount + */ + public @NonNull Builder setMaxSkippedAllocationCount(int value) { + mMaxSkippedAllocationCount = value; + return this; + } + + /** + * @see Options#percentageOfLeftSideGuard + */ + public int getPercentageOfLeftSideGuard() { + return mPercentageOfLeftSideGuard; + } + + /** + * @see Options#percentageOfLeftSideGuard + */ + public @NonNull Builder setPercentageOfLeftSideGuard(int value) { + mPercentageOfLeftSideGuard = value; + return this; + } + + /** + * @see Options#perfectRightSideGuard + */ + public boolean isPerfectRightSideGuard() { + return mPerfectRightSideGuard; + } + + /** + * @see Options#perfectRightSideGuard + */ + public @NonNull Builder setIsPerfectRightSideGuard(boolean value) { + mPerfectRightSideGuard = value; + return this; + } + + /** + * @see Options#ignoreOverlappedReading + */ + public boolean isIgnoreOverlappedReading() { + return mIgnoreOverlappedReading; + } + + /** + * @see Options#ignoreOverlappedReading + */ + public @NonNull Builder setIsIgnoreOverlappedReading(boolean value) { + mIgnoreOverlappedReading = value; + return this; + } + + /** + * @see Options#issueDumpFilePath + */ + public @Nullable String getIssueDumpFileDir() { + return mIssueDumpFileDir; + } + + /** + * @see Options#issueDumpFilePath + */ + public @NonNull Builder setIssueDumpFileDir(@Nullable String value) { + mIssueDumpFileDir = value; + return this; + } + + /** + * @see Options#targetSOPatterns + */ + public @NonNull List getTargetSOPatterns() { + return Collections.unmodifiableList(mTargetSOPatterns); + } + + /** + * @see Options#targetSOPatterns + */ + public @NonNull Builder setTargetSOPattern(@NonNull String value, String... nextValues) { + mTargetSOPatterns.clear(); + mTargetSOPatterns.add(value); + mTargetSOPatterns.addAll(Arrays.asList(nextValues)); + return this; + } + + /** + * @see Options#ignoredSOPatterns + */ + public @NonNull List getIgnoredSOPatterns() { + return Collections.unmodifiableList(mIgnoredSOPatterns); + } + + /** + * @see Options#ignoredSOPatterns + */ + public @NonNull Builder setIgnoredSOPattern(@NonNull String value, String... nextValues) { + mIgnoredSOPatterns.clear(); + mIgnoredSOPatterns.add(value); + mIgnoredSOPatterns.addAll(Arrays.asList(nextValues)); + return this; + } + + public @NonNull Options build() { + final Options res = new Options(); + if (getTargetSOPatterns().isEmpty()) { + setTargetSOPattern(DEFAULT_TARGET_SO_PATTERN); + } + res.maxAllocationSize = getMaxAllocationSize(); + res.maxDetectableAllocationCount = getMaxDetectableAllocationCount(); + res.maxSkippedAllocationCount = getMaxSkippedAllocationCount(); + res.percentageOfLeftSideGuard = getPercentageOfLeftSideGuard(); + res.perfectRightSideGuard = isPerfectRightSideGuard(); + res.ignoreOverlappedReading = isIgnoreOverlappedReading(); + res.issueDumpFilePath = generateIssueDumpFilePath(mContext, getIssueDumpFileDir()); + res.targetSOPatterns = getTargetSOPatterns().toArray(new String[0]); + res.ignoredSOPatterns = getIgnoredSOPatterns().toArray(new String[0]); + return res; + } + } + } + + private static String getDefaultIssueDumpDir(@NonNull Context context) { + final File result = new File(context.getCacheDir(), "memguard"); + return result.getAbsolutePath(); + } + + private static String getProcessSuffix(@NonNull Context context) { + final ActivityManager am = (ActivityManager) context.getSystemService(Context.ACTIVITY_SERVICE); + final List runningProcs = am.getRunningAppProcesses(); + final int myUid = Process.myUid(); + final int myPid = Process.myPid(); + for (ActivityManager.RunningAppProcessInfo procInfo : runningProcs) { + if (procInfo.uid == myUid && procInfo.pid == myPid) { + final int colIdx = procInfo.processName.lastIndexOf(':'); + if (colIdx >= 0) { + return procInfo.processName.substring(colIdx + 1); + } else { + return "main"; + } + } + } + return "@"; + } + + private static String generateIssueDumpFilePath(Context context, String dirPath) { + return new File(dirPath, "memguard_issue_in_proc_" + getProcessSuffix(context) + ".txt").getAbsolutePath(); + } +} diff --git a/matrix/matrix-android/matrix-opengl-leak/src/main/cpp/my_functions.h b/matrix/matrix-android/matrix-opengl-leak/src/main/cpp/my_functions.h index 4b14327d3..e0e87beb8 100644 --- a/matrix/matrix-android/matrix-opengl-leak/src/main/cpp/my_functions.h +++ b/matrix/matrix-android/matrix-opengl-leak/src/main/cpp/my_functions.h @@ -10,6 +10,8 @@ #include #include "BacktraceDefine.h" #include "Backtrace.h" +#include +#include #ifndef OPENGL_API_HOOK_MY_FUNCTIONS_H #define OPENGL_API_HOOK_MY_FUNCTIONS_H @@ -17,6 +19,7 @@ using namespace std; #define MEMHOOK_BACKTRACE_MAX_FRAMES MAX_FRAME_SHORT +#define RENDER_THREAD_NAME "RenderThread" static System_GlNormal_TYPE system_glGenTextures = NULL; static System_GlNormal_TYPE system_glDeleteTextures = NULL; @@ -64,6 +67,23 @@ void thread_id_to_string(thread::id thread_id, char *&result) { strcpy(result, stream.str().c_str()); } +inline void get_thread_name(char * thread_name) { + prctl(PR_GET_NAME, (char *) (thread_name)); +} + +inline bool is_render_thread() { + bool result = false; + char* thread_name = static_cast(malloc(BUF_SIZE)); + get_thread_name(thread_name); + if(strcmp(RENDER_THREAD_NAME, thread_name) == 0) { + result = true; + } + if(thread_name != nullptr) { + free(thread_name); + } + return result; +} + JNIEnv *GET_ENV() { JNIEnv *env; int ret = m_java_vm->GetEnv(reinterpret_cast(&env), JNI_VERSION_1_6); @@ -126,6 +146,10 @@ GL_APICALL void GL_APIENTRY my_glGenTextures(GLsizei n, GLuint *textures) { if (NULL != system_glGenTextures) { system_glGenTextures(n, textures); + if (is_render_thread()) { + return; + } + JNIEnv *env = GET_ENV(); int *result = new int[n]; @@ -183,6 +207,10 @@ GL_APICALL void GL_APIENTRY my_glDeleteTextures(GLsizei n, GLuint *textures) { if (NULL != system_glDeleteTextures) { system_glDeleteTextures(n, textures); + if (is_render_thread()) { + return; + } + JNIEnv *env = GET_ENV(); int *result = new int[n]; @@ -211,6 +239,10 @@ GL_APICALL void GL_APIENTRY my_glGenBuffers(GLsizei n, GLuint *buffers) { if (NULL != system_glGenBuffers) { system_glGenBuffers(n, buffers); + if (is_render_thread()) { + return; + } + JNIEnv *env = GET_ENV(); int *result = new int[n]; @@ -267,6 +299,10 @@ GL_APICALL void GL_APIENTRY my_glDeleteBuffers(GLsizei n, GLuint *buffers) { if (NULL != system_glDeleteBuffers) { system_glDeleteBuffers(n, buffers); + if (is_render_thread()) { + return; + } + JNIEnv *env = GET_ENV(); int *result = new int[n]; @@ -295,6 +331,10 @@ GL_APICALL void GL_APIENTRY my_glGenFramebuffers(GLsizei n, GLuint *buffers) { if (NULL != system_glGenFramebuffers) { system_glGenFramebuffers(n, buffers); + if (is_render_thread()) { + return; + } + JNIEnv *env = GET_ENV(); int *result = new int[n]; @@ -351,6 +391,10 @@ GL_APICALL void GL_APIENTRY my_glDeleteFramebuffers(GLsizei n, GLuint *buffers) if (NULL != system_glDeleteFramebuffers) { system_glDeleteFramebuffers(n, buffers); + if (is_render_thread()) { + return; + } + JNIEnv *env = GET_ENV(); int *result = new int[n]; @@ -380,6 +424,10 @@ GL_APICALL void GL_APIENTRY my_glGenRenderbuffers(GLsizei n, GLuint *buffers) { if (NULL != system_glGenRenderbuffers) { system_glGenRenderbuffers(n, buffers); + if (is_render_thread()) { + return; + } + JNIEnv *env = GET_ENV(); int *result = new int[n]; @@ -436,6 +484,10 @@ GL_APICALL void GL_APIENTRY my_glDeleteRenderbuffers(GLsizei n, GLuint *buffers) if (NULL != system_glDeleteRenderbuffers) { system_glDeleteRenderbuffers(n, buffers); + if (is_render_thread()) { + return; + } + JNIEnv *env = GET_ENV(); int *result = new int[n]; diff --git a/matrix/matrix-android/matrix-opengl-leak/src/main/java/com/tencent/matrix/openglleak/statistics/OpenGLResRecorder.java b/matrix/matrix-android/matrix-opengl-leak/src/main/java/com/tencent/matrix/openglleak/statistics/OpenGLResRecorder.java index cbb107e18..ddac6fdfe 100644 --- a/matrix/matrix-android/matrix-opengl-leak/src/main/java/com/tencent/matrix/openglleak/statistics/OpenGLResRecorder.java +++ b/matrix/matrix-android/matrix-opengl-leak/src/main/java/com/tencent/matrix/openglleak/statistics/OpenGLResRecorder.java @@ -156,11 +156,11 @@ public void setLeak(OpenGLInfo info) { break; } - if (isNeedIgnore(info)) { - item.release(); - iterator.remove(); - continue; - } +// if (isNeedIgnore(info)) { +// item.release(); +// iterator.remove(); +// continue; +// } if ((item.getType() == info.getType()) && (item.getThreadId().equals(info.getThreadId())) && (item.getId() == info.getId())) { if (mListener != null) { @@ -182,9 +182,9 @@ public void setMaybeLeak(OpenGLInfo info) { break; } - if (isNeedIgnore(info)) { - continue; - } +// if (isNeedIgnore(info)) { +// continue; +// } if ((item.getType() == info.getType()) && (item.getThreadId().equals(info.getThreadId())) && (item.getId() == info.getId())) { item.setMaybeLeak(true); diff --git a/matrix/matrix-android/matrix-resource-canary/matrix-resource-canary-common/src/main/java/com/tencent/matrix/resource/common/utils/StreamUtil.java b/matrix/matrix-android/matrix-resource-canary/matrix-resource-canary-common/src/main/java/com/tencent/matrix/resource/common/utils/StreamUtil.java index 3fa0f6110..506d9ad42 100644 --- a/matrix/matrix-android/matrix-resource-canary/matrix-resource-canary-common/src/main/java/com/tencent/matrix/resource/common/utils/StreamUtil.java +++ b/matrix/matrix-android/matrix-resource-canary/matrix-resource-canary-common/src/main/java/com/tencent/matrix/resource/common/utils/StreamUtil.java @@ -49,7 +49,25 @@ public static void closeQuietly(Object target) { } } + public static boolean preventZipSlip(java.io.File output, String zipEntryName) { + + try { + if (zipEntryName.contains("..") && new File(output, zipEntryName).getCanonicalPath().startsWith(output.getCanonicalPath() + File.separator)) { + return true; + } + } catch (IOException e) { + e.printStackTrace(); + return true; + } + return false; + } + public static void extractZipEntry(ZipFile zipFile, ZipEntry targetEntry, File output) throws IOException { + + if (preventZipSlip(output, targetEntry.getName())) { + throw new IllegalStateException("extractZipEntry entry " + targetEntry.getName() + " failed!"); + } + InputStream is = null; OutputStream os = null; try { diff --git a/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/config/TraceConfig.java b/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/config/TraceConfig.java index fc41c1735..96c90da9c 100644 --- a/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/config/TraceConfig.java +++ b/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/config/TraceConfig.java @@ -52,6 +52,8 @@ public class TraceConfig implements IDefaultConfig { public String anrTraceFilePath = ""; public String printTraceFilePath = ""; public boolean isHasActivity; + public boolean historyMsgRecorder; + public boolean denseMsgTracer; private TraceConfig() { this.isHasActivity = true; @@ -69,6 +71,8 @@ public String toString() { ss.append("* defaultStartupEnable:\t").append(defaultStartupEnable).append("\n"); ss.append("* defaultAnrEnable:\t").append(defaultAnrEnable).append("\n"); ss.append("* splashActivities:\t").append(splashActivities).append("\n"); + ss.append("* historyMsgRecorder:\t").append(historyMsgRecorder).append("\n"); + ss.append("* denseMsgTracer:\t").append(denseMsgTracer).append("\n"); return ss.toString(); } @@ -141,6 +145,15 @@ public String getPrintTraceFilePath() { return printTraceFilePath; } + @Override + public boolean isHistoryMsgRecorderEnable() { + return historyMsgRecorder; + } + + @Override + public boolean isDenseMsgTracerEnable() { + return denseMsgTracer; + } public Set getSplashActivities() { if (null == splashActivitiesSet) { @@ -300,7 +313,15 @@ public Builder enableMainThreadPriorityTrace(boolean enable) { return this; } + public Builder enableHistoryMsgRecorder(boolean enable) { + config.historyMsgRecorder = enable; + return this; + } + public Builder enableDenseMsgTracer(boolean enable) { + config.denseMsgTracer = enable; + return this; + } public TraceConfig build() { return config; } diff --git a/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/core/LooperMonitor.java b/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/core/LooperMonitor.java index 7752c519a..9e78ef7fd 100644 --- a/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/core/LooperMonitor.java +++ b/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/core/LooperMonitor.java @@ -17,6 +17,8 @@ package com.tencent.matrix.trace.core; import android.os.Build; +import android.os.Handler; +import android.os.HandlerThread; import android.os.Looper; import android.os.MessageQueue; import android.os.SystemClock; @@ -26,22 +28,49 @@ import android.util.Log; import android.util.Printer; +import com.tencent.matrix.util.MatrixHandlerThread; import com.tencent.matrix.util.MatrixLog; import com.tencent.matrix.util.ReflectUtils; import java.util.HashSet; import java.util.Map; import java.util.Objects; +import java.util.Queue; import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentLinkedQueue; public class LooperMonitor implements MessageQueue.IdleHandler { private static final String TAG = "Matrix.LooperMonitor"; private static final Map sLooperMonitorMap = new ConcurrentHashMap<>(); private static final LooperMonitor sMainMonitor = LooperMonitor.of(Looper.getMainLooper()); + private static final HandlerThread historyMsgHandlerThread = MatrixHandlerThread.getNewHandlerThread("historyMsgHandlerThread", HandlerThread.NORM_PRIORITY); + private static final Handler historyMsgHandler = new Handler(historyMsgHandlerThread.getLooper()); + private static long messageStartTime = 0; + private static final int HISTORY_QUEUE_MAX_SIZE = 200; + private static final int RECENT_QUEUE_MAX_SIZE = 5000; + + private static final Queue anrHistoryMQ = new ConcurrentLinkedQueue<>(); + private static final Queue recentMsgQ = new ConcurrentLinkedQueue<>(); + + private static String latestMsgLog = ""; + private static long recentMCount = 0; + private static long recentMDuration = 0; + public abstract static class LooperDispatchListener { boolean isHasDispatchStart = false; + boolean historyMsgRecorder = false; + boolean denseMsgTracer = false; + + public LooperDispatchListener(boolean historyMsgRecorder, boolean denseMsgTracer) { + this.historyMsgRecorder = historyMsgRecorder; + this.denseMsgTracer = denseMsgTracer; + } + + public LooperDispatchListener() { + + } public boolean isValid() { return false; @@ -241,16 +270,81 @@ public void println(String x) { } } + private static void recordMsg(final String log, final long duration, boolean denseMsgTracer) { + historyMsgHandler.post(new Runnable() { + @Override + public void run() { + enqueueHistoryMQ(new M(log, duration)); + } + }); + + if (denseMsgTracer) { + historyMsgHandler.post(new Runnable() { + @Override + public void run() { + enqueueRecentMQ(new M(log, duration)); + } + }); + } + } + + private static void enqueueRecentMQ(M m) { + if (recentMsgQ.size() == RECENT_QUEUE_MAX_SIZE) { + recentMsgQ.poll(); + } + recentMsgQ.offer(m); + + recentMDuration += m.d; + } + + private static void enqueueHistoryMQ(M m) { + if (anrHistoryMQ.size() == HISTORY_QUEUE_MAX_SIZE) { + anrHistoryMQ.poll(); + } + anrHistoryMQ.offer(m); + } + + public static Queue getHistoryMQ() { + enqueueHistoryMQ(new M(latestMsgLog, System.currentTimeMillis() - messageStartTime)); + return anrHistoryMQ; + } + + public static Queue getRecentMsgQ() { + return recentMsgQ; + } + + public static void cleanRecentMQ() { + recentMsgQ.clear(); + recentMCount = 0; + recentMDuration = 0; + } + + public static long getRecentMCount() { + return recentMCount; + } + + public static long getRecentMDuration() { + return recentMDuration; + } + private void dispatch(boolean isBegin, String log) { synchronized (listeners) { for (LooperDispatchListener listener : listeners) { if (listener.isValid()) { if (isBegin) { if (!listener.isHasDispatchStart) { + if (listener.historyMsgRecorder) { + messageStartTime = System.currentTimeMillis(); + latestMsgLog = log; + recentMCount++; + } listener.onDispatchStart(log); } } else { if (listener.isHasDispatchStart) { + if (listener.historyMsgRecorder) { + recordMsg(log, System.currentTimeMillis() - messageStartTime, listener.denseMsgTracer); + } listener.onDispatchEnd(log); } } @@ -260,4 +354,18 @@ private void dispatch(boolean isBegin, String log) { } } } + + public static class M { + public String l; + public long d; + M(String l, long d) { + this.l = l; + this.d = d; + } + + @Override + public String toString() { + return "{" + l + " -> " + d + '}'; + } + } } diff --git a/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/core/UIThreadMonitor.java b/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/core/UIThreadMonitor.java index 29d2b19da..55a8aaf44 100644 --- a/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/core/UIThreadMonitor.java +++ b/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/core/UIThreadMonitor.java @@ -117,8 +117,10 @@ public void init(TraceConfig config) { } vsyncReceiver = ReflectUtils.reflectObject(choreographer, "mDisplayEventReceiver", null); frameIntervalNanos = ReflectUtils.reflectObject(choreographer, "mFrameIntervalNanos", Constants.DEFAULT_FRAME_DURATION); + boolean historyMsgRecorder = config.historyMsgRecorder; + boolean denseMsgTracer = config.denseMsgTracer; - LooperMonitor.register(new LooperMonitor.LooperDispatchListener() { + LooperMonitor.register(new LooperMonitor.LooperDispatchListener(historyMsgRecorder, denseMsgTracer) { @Override public boolean isValid() { return isAlive; diff --git a/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/listeners/IDefaultConfig.java b/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/listeners/IDefaultConfig.java index 8ba6e8b27..2694439ee 100644 --- a/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/listeners/IDefaultConfig.java +++ b/matrix/matrix-android/matrix-trace-canary/src/main/java/com/tencent/matrix/trace/listeners/IDefaultConfig.java @@ -46,4 +46,8 @@ public interface IDefaultConfig { String getPrintTraceFilePath(); + boolean isHistoryMsgRecorderEnable(); + + boolean isDenseMsgTracerEnable(); + } diff --git a/samples/sample-android/build.gradle b/samples/sample-android/build.gradle index 2e2e83265..d49ef8645 100644 --- a/samples/sample-android/build.gradle +++ b/samples/sample-android/build.gradle @@ -7,7 +7,7 @@ buildscript { buildToolsVersion = '29.0.2' javaVersion = JavaVersion.VERSION_1_8 - MATRIX_VERSION = "2.0.1" + MATRIX_VERSION = "2.0.2" GROUP = 'com.tencent.matrix' VERSION_NAME = "${MATRIX_VERSION}"