Merge pull request #660 from Tencent/feature/dev

Matrix v2.0.2

Author: gryamy

README.md

@@ -1,7 +1,7 @@
 ![Matrix-icon](assets/img/readme/header.png)
 [![license](http://img.shields.io/badge/license-BSD3-brightgreen.svg?style=flat)](https://github.com/Tencent/matrix/blob/master/LICENSE)
 [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/Tencent/matrix/pulls)
-[![WeChat Approved](https://img.shields.io/badge/Wechat%20Approved-2.0.1-red.svg)](https://github.com/Tencent/matrix/wiki)
+[![WeChat Approved](https://img.shields.io/badge/Wechat%20Approved-2.0.2-red.svg)](https://github.com/Tencent/matrix/wiki)
 [![CircleCI](https://circleci.com/gh/Tencent/matrix.svg?style=shield)](https://app.circleci.com/pipelines/github/Tencent/matrix)
 
 (中文版本请参看[这里](#matrix_cn))  
@@ -140,6 +140,10 @@ At this point, Matrix has been integrated into the app and is beginning to colle
 - **Battery Canary:**
 
   App thread activities monitor (Background watch & foreground loop watch), Sonsor usage monitor (WakeLock/Alarm/Gps/Wifi/Bluetooth), Background network activities (Wifi/Mobile) monitor.
+  
+- **MemGuard**
+
+  Detect heap memory overlap, use-after-free and double free issues.
 
 
 ## Features
@@ -197,6 +201,14 @@ At this point, Matrix has been integrated into the app and is beginning to colle
 + **Non-invasive.** It is based on PLT-hook([iqiyi/xHook](https://github.com/iqiyi/xHook)), so we do NOT need to recompile the native libraries.
 + WebView still works after using this tool.
 
+#### MemGuard
+
++ A tool base on GWP-Asan to detect heap memory issues.
++ **Non-invasive.** It is based on PLT-hook([iqiyi/xHook](https://github.com/iqiyi/xHook)), so we do NOT need to recompile the native libraries.
++ It's able to apply on specific libraries that needs to be detected by RegEx.
+
++ It detects heap memory accessing overlap, use-after-free and double free issues.
+
 
 #### Backtrace Component
 
@@ -208,7 +220,7 @@ At this point, Matrix has been integrated into the app and is beginning to colle
 
 1. Configure `MATRIX_VERSION` in gradle.properties.
 ``` gradle
-  MATRIX_VERSION=2.0.1
+  MATRIX_VERSION=2.0.2
 ```
 
 2. Add `matrix-gradle-plugin` in your build.gradle:
@@ -357,11 +369,11 @@ Then other components in Matrix could use Quikcen Backtrace to unwind stacktrace
 
 #### APK Checker Usage
 
-APK Checker can run independently in Jar ([matrix-apk-canary-2.0.1.jar](https://repo.maven.apache.org/maven2/com/tencent/matrix/matrix-apk-canary/2.0.1/matrix-apk-canary-2.0.1.jar)）  mode, usage:
+APK Checker can run independently in Jar ([matrix-apk-canary-2.0.2.jar](https://repo.maven.apache.org/maven2/com/tencent/matrix/matrix-apk-canary/2.0.2/matrix-apk-canary-2.0.2.jar)）  mode, usage:
 
 
 ```shell
-java -jar matrix-apk-canary-2.0.1.jar
+java -jar matrix-apk-canary-2.0.2.jar
 Usages: 
     --config CONFIG-FILE-PATH
 or
@@ -420,7 +432,7 @@ Matrix is under the BSD license. See the [LICENSE](https://github.com/Tencent/Ma
 
 # <a name="matrix_cn">Matrix</a>
 ![Matrix-icon](assets/img/readme/header.png)
-[![license](http://img.shields.io/badge/license-BSD3-brightgreen.svg?style=flat)](https://github.com/Tencent/matrix/blob/master/LICENSE)[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/Tencent/matrix/pulls)  [![WeChat Approved](https://img.shields.io/badge/Wechat%20Approved-2.0.1-red.svg)](https://github.com/Tencent/matrix/wiki)
+[![license](http://img.shields.io/badge/license-BSD3-brightgreen.svg?style=flat)](https://github.com/Tencent/matrix/blob/master/LICENSE)[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg)](https://github.com/Tencent/matrix/pulls)  [![WeChat Approved](https://img.shields.io/badge/Wechat%20Approved-2.0.2-red.svg)](https://github.com/Tencent/matrix/wiki)
 
 **Matrix** 是一款微信研发并日常使用的应用性能接入框架，支持iOS, macOS和Android。
 Matrix 通过接入各种性能监控方案，对性能监控项的异常数据进行采集和分析，输出相应的问题分析、定位与优化建议，从而帮助开发者开发出更高质量的应用。
@@ -529,16 +541,25 @@ curBuilder.pluginListener = <一个遵循 MatrixPluginListenerDelegate 的对象
 Matrix-android 当前监控范围包括：应用安装包大小，帧率变化，启动耗时，卡顿，慢方法，SQLite 操作优化，文件读写，内存泄漏等等。
 - APK Checker:
   针对 APK 安装包的分析检测工具，根据一系列设定好的规则，检测 APK 是否存在特定的问题，并输出较为详细的检测结果报告，用于分析排查问题以及版本追踪
+  
 - Resource Canary:
   基于 WeakReference 的特性和 [Square Haha](https://github.com/square/haha) 库开发的 Activity 泄漏和 Bitmap 重复创建检测工具
+  
 - Trace Canary:
   监控ANR、界面流畅性、启动耗时、页面切换耗时、慢函数及卡顿等问题
+  
 - SQLite Lint:
   按官方最佳实践自动化检测 SQLite 语句的使用质量
+  
 - IO Canary:
   检测文件 IO 问题，包括：文件 IO 监控和 Closeable Leak 监控
+  
 - Battery Canary:
   监控 App 活跃线程（待机状态 & 前台 Loop 监控）、ASM 调用 (WakeLock/Alarm/Gps/Wifi/Bluetooth 等传感器)、 后台流量 (Wifi/移动网络)等 Battery Historian 统计 App 耗电的数据
+  
+- MemGuard
+
+  检测堆内存访问越界、使用释放后的内存、重复释放等问题
 
 ## 特性
 
@@ -599,6 +620,13 @@ Matrix-android 当前监控范围包括：应用安装包大小，帧率变化
 + 无侵入，基于 PLT-hook([iqiyi/xHook](https://github.com/iqiyi/xHook))，无需重编 native 库
 + 使用该工具后 WebView 仍可正常工作
 
+#### MemGuard
+
++ 一个基于 GWP-Asan 修改的堆内存问题检测工具
++ 无侵入，基于 PLT-hook([iqiyi/xHook](https://github.com/iqiyi/xHook))，无需重编 native 库
++ 可根据正则表达式指定被检测的目标库
++ 可检测堆内存访问越界、使用释放后的内存和双重释放等问题
+
 #### Backtrace Component
 - 基于 DWARF 以及 ARM 异常处理数据进行简化并生成全新的 quicken unwind tables 数据，用于实现可快速回溯 native 调用栈的 backtrace 组件。回溯速度约是 libunwindstack 的 15x ~ 30x 左右。
 
@@ -608,7 +636,7 @@ Matrix-android 当前监控范围包括：应用安装包大小，帧率变化
 
 1. 在你项目根目录下的 gradle.properties 中配置要依赖的 Matrix 版本号，如：
 ``` gradle
-  MATRIX_VERSION=2.0.1
+  MATRIX_VERSION=2.0.2
 ```
 
 2. 在你项目根目录下的 build.gradle 文件添加 Matrix 依赖，如：
@@ -754,10 +782,10 @@ WeChatBacktrace.instance().configure(getApplicationContext()).commit();
 
 #### APK Checker
 
-APK Check 以独立的 jar 包提供 ([matrix-apk-canary-2.0.1.jar](https://repo.maven.apache.org/maven2/com/tencent/matrix/matrix-apk-canary/2.0.1/matrix-apk-canary-2.0.1.jar)），你可以运行：
+APK Check 以独立的 jar 包提供 ([matrix-apk-canary-2.0.2.jar](https://repo.maven.apache.org/maven2/com/tencent/matrix/matrix-apk-canary/2.0.2/matrix-apk-canary-2.0.2.jar)），你可以运行：
 
 ```cmd
-java -jar matrix-apk-canary-2.0.1.jar
+java -jar matrix-apk-canary-2.0.2.jar
 ```
 
 查看 Usages 来使用它。

matrix/matrix-android/gradle.properties

@@ -14,7 +14,7 @@ org.gradle.jvmargs=-Xmx2048m -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryErro
 # org.gradle.parallel=true
 #Tue Jun 20 10:24:33 CST 2017
 
-VERSION_NAME_PREFIX=2.0.1
+VERSION_NAME_PREFIX=2.0.2
 VERSION_NAME_SUFFIX=
 ## two options: Internal (for wechat),  External (for public repo)
 PUBLISH_CHANNEL=Internal

matrix/matrix-android/matrix-apk-canary/src/main/java/com/tencent/matrix/apk/model/task/UnzipTask.java

@@ -227,11 +227,16 @@ private String reverseResguard(String dirName, String filename) {
 
     private String writeEntry(ZipFile zipFile, ZipEntry entry) throws IOException {
 
+        String entryName = entry.getName();
+        if (Util.preventZipSlip(outputFile, entryName)) {
+            Log.e(TAG, "writeEntry entry %s failed!", entryName);
+            return null;
+        }
+
         int readSize;
         byte[] readBuffer = new byte[4096];
         BufferedOutputStream bufferedOutput = null;
         InputStream zipInputStream = null;
-        String entryName = entry.getName();
         String outEntryName = null;
         String filename;
         File dir;

matrix/matrix-android/matrix-backtrace/build.gradle

@@ -19,6 +19,7 @@ android {
 
         externalNativeBuild {
             cmake {
+                targets = ['wechatbacktrace', 'unwindstack']
                 arguments = ['-DANDROID_STL=c++_shared',
                              "-DEnableLOG=${gradle.enableLog() ? "ON" : "OFF"}" as String,
                              "-DQUT_STATISTIC_ENABLE=${gradle.enableLog() ? "ON" : "OFF"}" as String]

matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/CMakeLists.txt

@@ -17,6 +17,7 @@ SET(
         ${CMAKE_CURRENT_SOURCE_DIR}/ElfInterfaceArm.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/JitDebug.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Log.cpp
+        ${CMAKE_CURRENT_SOURCE_DIR}/LocalUnwinder.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/MapInfo.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Maps.cpp
         ${CMAKE_CURRENT_SOURCE_DIR}/Memory.cpp

matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/LocalUnwinder.cpp

@@ -31,7 +31,9 @@
 
 #include <memory>
 #include <string>
+#include <sstream>
 #include <vector>
+#include <iomanip>
 
 #include <unwindstack/Elf.h>
 #include <unwindstack/LocalUnwinder.h>
@@ -142,4 +144,105 @@ bool LocalUnwinder::Unwind(std::vector<LocalFrameData>* frame_info, size_t max_f
   return num_frames != 0;
 }
 
+size_t LocalUnwinder::UnwindImpl(std::unique_ptr<unwindstack::Regs>& regs, void** pcs, size_t max_frames) {
+  ArchEnum arch = regs->Arch();
+
+  size_t num_frames = 0;
+  bool adjust_pc = false;
+  while (true) {
+    uint64_t cur_pc = regs->pc();
+    uint64_t cur_sp = regs->sp();
+
+    MapInfo* map_info = GetMapInfo(cur_pc);
+    if (map_info == nullptr) {
+      break;
+    }
+
+    Elf* elf = map_info->GetElf(process_memory_, arch);
+    uint64_t rel_pc = elf->GetRelPc(cur_pc, map_info);
+    uint64_t step_pc = rel_pc;
+    uint64_t pc_adjustment;
+    if (adjust_pc) {
+      pc_adjustment = GetPcAdjustment(rel_pc, elf, arch);
+    } else {
+      pc_adjustment = 0;
+    }
+    step_pc -= pc_adjustment;
+
+    bool finished = false;
+    if (elf->StepIfSignalHandler(rel_pc, regs.get(), process_memory_.get())) {
+      step_pc = rel_pc;
+    } else if (!elf->Step(step_pc, regs.get(), process_memory_.get(), &finished)) {
+      finished = true;
+    }
+
+    // Skip any locations that are within this library.
+    if (num_frames != 0 || !ShouldSkipLibrary(map_info->name)) {
+      pcs[num_frames++] = (void*) cur_pc;
+      if (num_frames >= max_frames) {
+        break;
+      }
+    }
+
+    if (finished || num_frames == max_frames ||
+        (cur_pc == regs->pc() && cur_sp == regs->sp())) {
+      break;
+    }
+    adjust_pc = true;
+  }
+  return num_frames;
+}
+
+size_t LocalUnwinder::Unwind(void** pcs, size_t max_frames) {
+  std::unique_ptr<unwindstack::Regs> regs(unwindstack::Regs::CreateFromLocal());
+  unwindstack::RegsGetLocal(regs.get());
+  return UnwindImpl(regs, pcs, max_frames);
+}
+
+size_t LocalUnwinder::Unwind(void* ucontext, void** pcs, size_t max_frames) {
+  std::unique_ptr<unwindstack::Regs> regs(unwindstack::Regs::CreateFromUcontext(Regs::CurrentArch(), ucontext));
+  return UnwindImpl(regs, pcs, max_frames);
+}
+
+extern "C" char* __cxa_demangle(const char*, char*, size_t*, int* status);
+
+bool LocalUnwinder::GetStackElementString(uint64_t pc, std::string* string_out) {
+  MapInfo* map_info = GetMapInfo(pc);
+  if (map_info == nullptr) {
+    return false;
+  }
+  auto elf = map_info->GetElf(process_memory_, Regs::CurrentArch());
+  uint64_t rel_pc = elf->GetRelPc(pc, map_info);
+
+  std::stringstream ss;
+  std::ios fmt(nullptr);
+  fmt.copyfmt(ss);
+  ss << "pc " << std::hex << std::setw(sizeof(void*) * 2) << std::setfill('0') << rel_pc << std::dec << ' ';
+  ss.copyfmt(fmt);
+
+  std::string funcName;
+  uint64_t funcOffset = 0;
+  bool isFuncNameGot = map_info->GetFunctionName(rel_pc, &funcName, &funcOffset);
+  if (isFuncNameGot) {
+    char* demangledFuncName = nullptr;
+    if (!funcName.empty()) {
+      demangledFuncName = __cxa_demangle(funcName.c_str(), nullptr, nullptr, nullptr);
+    }
+    if (demangledFuncName != nullptr) {
+      ss << map_info->name << " (" << demangledFuncName << ")";
+      free(demangledFuncName);
+    } else {
+      ss << map_info->name << " (" << funcName << ")";
+    }
+  } else {
+    ss << map_info->name << " (?\?\?)";
+  }
+  auto buildID = map_info->GetPrintableBuildID();
+  if (!buildID.empty()) {
+    ss << " (BuildID: " << map_info->GetPrintableBuildID() << ")";
+  }
+  *string_out = ss.str();
+  return true;
+}
+
 }  // namespace unwindstack

matrix/matrix-android/matrix-backtrace/src/main/cpp/external/libunwindstack/include/unwindstack/LocalUnwinder.h

@@ -66,14 +66,22 @@ class LocalUnwinder {
 
   bool Unwind(std::vector<LocalFrameData>* frame_info, size_t max_frames);
 
+  size_t Unwind(void** pcs, size_t max_frames);
+
+  size_t Unwind(void* ucontext, void** pcs, size_t max_frames);
+
   bool ShouldSkipLibrary(const std::string& map_name);
 
   MapInfo* GetMapInfo(uint64_t pc);
 
+  bool GetStackElementString(uint64_t pc, std::string* string_out);
+
   ErrorCode LastErrorCode() { return last_error_.code; }
   uint64_t LastErrorAddress() { return last_error_.address; }
 
  private:
+  size_t UnwindImpl(std::unique_ptr<unwindstack::Regs>& regs, void** pcs, size_t max_frames);
+
   pthread_rwlock_t maps_rwlock_;
   std::unique_ptr<LocalUpdatableMaps> maps_ = nullptr;
   std::shared_ptr<Memory> process_memory_;

matrix/matrix-android/matrix-backtrace/src/main/cpp/libwechatbacktrace/Dummy.cpp

@@ -0,0 +1,4 @@
+//
+// Created by YinSheng Tang on 2021/9/23.
+//
+

matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpDelegate.java

@@ -27,6 +27,7 @@
 import android.os.Message;
 import android.os.OperationCanceledException;
 import android.os.Process;
+import android.system.ErrnoException;
 import android.system.Os;
 import android.system.StructStat;
 import android.util.Pair;
@@ -518,8 +519,13 @@ public void run() {
                     iterateTargetDirectory(file, cs, new FileFilter() {
                         @Override
                         public boolean accept(File pathname) {
-                            count[0] += 1;
-                            count[1] += pathname.isFile() ? pathname.length() : 0;
+                            try {
+                                StructStat stat = Os.lstat(pathname.getAbsolutePath());
+                                count[0] += 1;
+                                count[1] += stat.st_blocks * stat.st_blksize;
+                            } catch (ErrnoException e) {
+                                MatrixLog.printErrStackTrace(TAG, e, "");
+                            }
                             return false;
                         }
                     });

matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpScheduler.java

@@ -197,10 +197,10 @@ private final static class IdleReceiver extends BroadcastReceiver {
         Handler mIdleHandler;
 
         Context mContext;
-        private WeChatBacktrace.WarmUpTiming mTiming;
-        private long mWarmUpDelay;
+        private final WeChatBacktrace.WarmUpTiming mTiming;
+        private final long mWarmUpDelay;
 
-        private Set<TaskType> mTasks = new HashSet<>();
+        private final Set<TaskType> mTasks = new HashSet<>();
 
         IdleReceiver(Context context, Handler idleHandler, WeChatBacktrace.WarmUpTiming timing,
                             long delay) {

matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WarmUpUtility.java

@@ -41,10 +41,10 @@ class WarmUpUtility {
     private final static String FILE_BLOCKED_LIST = "blocked-list";
     private final static String FILE_UNFINISHED = "unfinished";
 
-    final static long DURATION_LAST_ACCESS_FAR_FUTURE = 30L * 24 * 3600 * 1000; // milliseconds
-    final static long DURATION_LAST_ACCESS_EXPIRED = 60L * 24 * 3600 * 1000; // milliseconds
-    final static long DURATION_CLEAN_UP_EXPIRED = 3L * 24 * 3600 * 1000; // milliseconds
-    final static long DURATION_CLEAN_UP = 3L * 24 * 3600 * 1000; // milliseconds
+    final static long DURATION_LAST_ACCESS_FAR_FUTURE = 7L * 24 * 3600 * 1000; // milliseconds
+    final static long DURATION_LAST_ACCESS_EXPIRED = 3L * 24 * 3600 * 1000; // milliseconds
+    final static long DURATION_CLEAN_UP_EXPIRED = 2L * 24 * 3600 * 1000; // milliseconds
+    final static long DURATION_CLEAN_UP = 2L * 24 * 3600 * 1000; // milliseconds
     final static long DURATION_DISK_USAGE_COMPUTATION = 3L * 24 * 3600 * 1000; // milliseconds
 
     final static int WARM_UP_FILE_MAX_RETRY = 3;
@@ -61,8 +61,7 @@ private static int retryCount(Context context, String key) {
                 mUnfinishedWarmUp = WarmUpUtility.readUnfinishedMaps(context);
             }
             Integer value = mUnfinishedWarmUp.get(key);
-            int retryCount = value != null ? value : 0;
-            return retryCount;
+            return value != null ? value : 0;
         }
 
         public static boolean check(Context context, String pathOfElf, int offset) {

matrix/matrix-android/matrix-backtrace/src/main/java/com/tencent/matrix/backtrace/WeChatBacktrace.java

@@ -75,8 +75,8 @@ public static String getBaseODEXPath(Context context) {
     private volatile boolean mInitialized;
     private volatile boolean mConfigured;
     private volatile Configuration mConfiguration;
-    private WarmUpDelegate mWarmUpDelegate = new WarmUpDelegate();
-    private Handler mHandler = new Handler(Looper.getMainLooper());
+    private final WarmUpDelegate mWarmUpDelegate = new WarmUpDelegate();
+    private final Handler mHandler = new Handler(Looper.getMainLooper());
 
     private static boolean sLibraryLoaded = false;
 
@@ -345,7 +345,7 @@ public final static class Configuration {
         String mPathOfXLogSo = null;
 
         private boolean mCommitted = false;
-        private WeChatBacktrace mWeChatBacktrace;
+        private final WeChatBacktrace mWeChatBacktrace;
 
         Configuration(Context context, WeChatBacktrace backtrace) {
             mContext = context;