PCB BlogHappy Coding2019-09-11T19:23:44.177Zhttp://bin2415.github.io/binpangHexopaths in linuxhttp://bin2415.github.io/2019/09/09/linux-path/2019-09-10T00:40:10.000Z2019-09-11T19:23:44.177ZThere are many “path” in linux. Such as executable search PATH, compiler include path, linker path, dynamic library search path LD_LIBRARY_PATH.
PATH
PATH is an environment variable. It defines the executable search path. When you run executable file without path(absolute or relative path), it will search the executable file from directories that defined by PATH.
At initial, the PATH is defined in /etc/environment.
We can append the PATH in .bashrc(if terminal is bash) or .zshrc(if terminal is zsh), so the appened PATH is avaiable whenever we open a new terminal.
Include Path
For gcc, it will look in several different places for headers[1]. It will look for headers requested with #include in default directories. We can check the directories by using gcc --verbose
1 2 3 4 5 6 7 8 9 10 11
$ cpp --verbose
#include "..." search starts here: #include <...> search starts here: /usr/lib/gcc/x86_64-linux-gnu/7/include /usr/local/include /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed /usr/x86_64-linux-gnu/include /usr/include/x86_64-linux-gnu /usr/include End of search list.
We can specify the include path when we compile the source code by adding -I/path/to/be/added.
Linker Search Path
In order to do the symbol resolution, linker will search for the unresolved symbols in linker search path.
We can use gcc -print-search-dirs | sed '/^lib/b 1;d;:1;s,/[^/.][^/]*/\.\./,/,;t 1;s,:[^=]*=,:;,;s,;,; ,g' | tr \; \\012 to print default linker search path:
We can specify the linker search path to gcc by adding -L/path/to/be/added.
We can use -B/path/to/directory of gcc to specify the gcc search path. It will insert the path in the head of the searh path list. For example, we add -B/usr/test/ path to gcc:
LD_LIBRARY_PATH is an environment variable you set to give the run-time shared library loader (ld.so) an extra set of directories to look for when searching for shared libraries[2].
]]>
<p>There are many “path” in linux. Such as executable search <code>PATH</code>, compiler include path, linker path, dynamic library search p
Ghidra Python Uses Other Packageshttp://bin2415.github.io/2019/08/22/ghidra-python/2019-08-22T13:49:08.000Z2019-08-22T14:10:54.908ZGhidra is a reverse engineering framework. It is similar to IDA pro and it is a open source project. Ghidra also provide java and python api to write plugin. Actually, its “python” is Jython, and its version is python2.7. And its site-packages location is /opt/ghidra/Ghidra/Features/Python/data/jython-2.7.1/Lib. It only supports some uses of basic packages. And I want to use other packages(such as protobuf) in Ghidra python, I did not find a proper way to install the protobuf in Ghidra Jython’s site-packages location.
And I find another way to solve it. Firstly, I install protobuf in my system’s default python2.7.
1
sudo pip2 install protobuf
The protobuf is installed into python2.7 site-packages location. To find the site-packges location, you can open python2.7 and type below:
1 2 3 4
In [1]: import site In [2]: print(site.getsitepackages())
]]>
<p>Install binaryninja python path.</p>
<p>Firstly, get the python default path:</p>
<figure class="highlight plain"><table><tr><td class="g
Be Careful When Modifing Binary Program(Abount Stack Alignment)http://bin2415.github.io/2019/07/12/stack-alignment/2019-07-12T14:53:28.000Z2019-07-12T15:18:32.303ZRecently, I want to modify the binary program which add a instruction push $rbx in the binary program. And it will increment the stack by 8 bytes. It sounds good for the binary program.
And I debug the modified binary with gdb, and find that it crashes in movaps instruction.
1
movaps xmmword ptr [rsp+0x50], xmm0
So what happend?
And I found the answers from some blogs(see reference). movaps is “move aligned packed single-precision floating-point values”. If the instruction’s operand is memory, the memory address must be aligned to 16 bytes. And I print the $rsp+0x50, its not 16 bytes alignment. That’s because I pushed $rbx into the stack and increment the rsp to 8 bytes, and that results in rsp is not aligned to 16 bytes.
]]>
<p>Recently, I want to modify the binary program which add a instruction <code>push $rbx</code> in the binary program. And it will increment
C Programs Before main Functionhttp://bin2415.github.io/2019/07/02/ctr-things/2019-07-02T14:35:48.000Z2019-07-02T15:00:27.177ZWhat does the program do before main() function
The picture from Patrick Horgan describes what will the c program do before main function.
crt0.o, ctri.o, ctrbegin.o, ctrn.o
If you compile a c program, the linker will link crt0.o, ctri.o, ctrbegin.o, ctrn.o with the target object together.
crt0.o contains _start function, it will initialize the process before call main function, it is defined in libc’s crt0.s file.
According to osdev, ctri.o defines the header of _init and _fini function, and ctrn.o defines the footer of _init and _fini function. And linker will link ctrbegin.o’s section .init and .fini between ctri.o and ctrn.o
ctrbegin.o also defines some functions such as deregister_tm_clones, register_tm_clones, __do_global_dtors_aux, frame_dummy
]]>
<h2 id="Add-a-remote-repo"><a href="#Add-a-remote-repo" class="headerlink" title="Add a remote repo"></a>Add a remote repo</h2><figure class
Compile Kernel using llvm/clanghttp://bin2415.github.io/2019/04/16/llvm-kernel/2019-04-16T20:20:07.000Z2019-07-12T15:29:21.033ZReference
git clone https://github.com/ramosian-glider/clang-kernel-build.git cd clang-kernel-build export WORLD=`pwd`
Install Clang from Chromium:
1 2 3 4 5 6 7
cd $WORLD # Instruction taken from http://llvm.org/docs/LibFuzzer.html mkdir TMP_CLANG cd TMP_CLANG git clone https://chromium.googlesource.com/chromium/src/tools/clang cd .. TMP_CLANG/clang/scripts/update.py
(To update Clang later on, do (cd TMP_CLANG/clang ; git pull) and run update.py again.)
Clone the linux source tree
1 2 3 4
cd $WORLD git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git cd linux-stable git reset --hard v4.16
(Note that kernel version v4.16 or older is fine, otherwise the latest clang lacks asm-goto support(llvm already support))ref1 and ref2。
Configure and build the kernel
1 2 3 4 5 6
cd $WORLD export CLANG_PATH=`pwd`/third_party/llvm-build/Release+Asserts/bin/ cd linux-stable make CC=$CLANG_PATH/clang defconfig make CC=$CLANG_PATH/clang kvmconfig make CC=$CLANG_PATH/clang -j64 2>&1 | tee build.log
Set up the VM
1 2 3 4
cd $WORLD wget https://raw.githubusercontent.com/google/sanitizers/master/address-sanitizer/kernel_buildbot/create_os_image.sh # create_os_image.sh requires sudo sh create_os_image.sh
Run the VM
1 2 3 4
cd $WORLD ./run_qemu.sh # in a separate console: ssh -i ssh/id_rsa -p 10023 root@localhost
in drivers/gpu/drm/amd/display/dc/cals/Makefile and drivers/gpu/drm/amd/display/dc/dml/Makefile file, they specify CFLAGS with mpreferred-stack-boundary, which is not supported by clang. Clang has the flag -mstack-alignment=4 that equals. So replace them with -mstack-alignment=4 in these two files.
# gdb config CONFIG_GDB_SCRIPTS=y CONFIG_DEBUG_INFO=y # CONFIG_DEBUG_INFO_REDUCED is not set # CONFIG_RANDOMIZE_BASE is not set
# kgdb config # CONFIG_STRICT_KERNEL_RWX is not set CONFIG_FRAME_POINTER=y CONFIG_KGDB=y CONFIG_KGDB_SERIAL_CONSOLE=y
# kdb config # CONFIG_STRICT_KERNEL_RWX is not set CONFIG_FRAME_POINTER=y CONFIG_KGDB=y CONFIG_KGDB_SERRIAL_CONSOLE=y CONFIG_KGDB_KDB=y CONFIG_KDB_KEYBOARD=y
# manually debug using the SysRq-G CONFIG_MAGIC_SysRq=y
Fuzzer: T-Fuzz uses an existing coverage guided fuzzer to generate inputs. T-Fuzz depends on the fuzzer to keep track of the paths taken by all the generated inputs and realtime status infomation regarding whether it is “stuck”. As output, the fuzzer produces all the generated inputs. Any identified crashing inputs are recorded for further anlysis.
Program Transformer: When the fuzzer gets “stuck”, T-Fuzz invokes its Program Transformer to generate tranformed programs. Using the inputs generated by the fuzzer, the Program Transformer first traces the program under test to detect the NCC candidates and then transforms copies of the program by removing certain detected NCC candidates.
Crash Analyzer: For crashing inputs found against the transformed programs, the Crash Analyser filters false positives using a symbolic-execution based analysis technique.
T-Fuzz Design
Detecting NCCs: NCCs are those sanity checks which are present in the program logic to filter some orthogonal data, e.g., the check for a magic value in the decompressor example above. NCCs can be removed without triggering spurious bugs as they are not intended to prevent bugs. This paper uses a lightweight method to find the NCCs. Firstly, they define the concept of boundary edges: the edges connecting the nodes that were covered by the fuzzer-generated inputs and those that were not. The method that find the NCCs in this paper is over-approximation, so they find two ways to prune undesired NCC condidates.
Program Transformation: After finding NCCs, T-Fuzz should “remove” the NCCs conditions to guide the execution to the another branch. T-Fuzz transforms programs by replacing the detected NCC candidates with negated conditional jump.
Filtering out False Positives and Reproducing Bugs: As the removed NCC candidates might be meaningful guards in the original program(as opposed to, e.g., magic number checks), removing detected NCC edges might introduce new bugs in the transformed program. Consequently, T-Fuzz’s Crash Analyzer verifies that each bug in the transformaed program is also present in the original proram, thus filtering out false positives. The Crash Analyser uses a transformation-aware combination of the preconstrained tracing technique leveraged by Driller and the Path Kneading techniques proposed by ShellSwap to collect path constraints of the original program by tracing the program path leading to a crash in the transformed program.
Context-sensitive branch coverage. AFL uses context-insensitive branch coverage to approximate program states. This paper adding context information to branch.
Scalable byte-level taint tracking. Most path constraints depend on only a few bytes in the input. By tracking which input bytes flow into each path constraint, Angora mutates only these bytes instead of the entire input, therefore reducing the space of exploration substantially.
Search based on gradient descent. When mutating the input to satisfy a path constraint. Angora avoids symbolic execution, which is expensive and cannot solve many types of constraints. Instead, Angora uses the gradient descent algorithm popular in machine learning to solve path constraints.
Type and shape inference. Many bytes in the input are used collectively as a single value in the program, e.g., a group of four bytes in the input used as a 32-bit signed integer in the program. To allow gradient descent to search efficiently, Angora locates the above group and infers its type.
Designing New Operating Primitives to Improve Fuzzing Performance(CCS 17)
Mutating inputs(1). AFL uses an evolutionary coverage-based mutation technique to generate test cases for discovering new execution paths of the target application. In AFL, an execution path is represented as a sequence of taken branches(i.e., a coverage bitmap) in the target instance for a given input. To track whether a branch is taken, AFL instruments every conditional branch and function entry of the target application at the time of compilation.
Launching the target application(2). Traditional fuzzers call fork() followed by execve() to launch an instance of the target application. This process occurs in every fuzzing loop to deliver a new input to the target application. It is not only time consuming, but also a non-scalable operation. Previous research shows that the majority of fuzzing execution explores only the shallow part of the code and terminates quickly(e.g., because of invalid input format), which results in frequent executions for the input cases. Thus, the cost of fork() and execve() dominates the cost of fuzzing. To mitigate this cost, AFL introduced a fork server, which is similar to a Zygote process in Android that eliminates the heavyweight execve() system call. After instantiating a target application, the fork server waits for a starting signal sent over the pip from the AFL instance. Upon receiving the request, it first clones the already-loaded program using fork() and the child process continues the execution of the original target code immediately from the entry point(i.e., main) with a given input generated for the current fuzzing loop. The parent process waits for the termination of its child, and then informs the AFL process. The AFL process collects the branch coverage of the past execution, and maintains the test input if it is interesting.
Bookkeeping results(3, 4). The fork server also initializes a shared memory(also known as tracing bitmap) between the AFL instance and the target application. The instance records all the coverage information during the execution and writes it to the shared tracing bitmap, which summarizes the branch coverage of the past execution.
Fuzzing in parallel(6). AFL also supports parallel fuzzing to completely utilize resources available on a multi-core machine and expedite the fuzzing process. In this case, each AFL instance independently executes without explicit contention among themselves(i.e., embarrassingly parallel). From the perspective of the design of AFL, the fuzzing operation should linearly scale with increasing core count. Moreover, to avoid apparent contention on file system accesses, each AFL instance works in its private working directory for test cases. At the end of a fuzzing loop, the AFL instance scans the output directories of other instances to learn their test cases, called the syncing phase. For each collaborating neighbor, it keeps a test case identifier, which indicates the last test case it has checked. It figures out all the test cases that have an identifier larger than the reserved one, and re-executes them one by one. If a test case covers a new path that has not been discovered by the instance itself, the test case is copied to its oen directory for further mutation.
QSYM: A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing(Usenix 18)
然而使用IR则引发了额外的overhead。首先，从机器指令到IR的转换本身就有overhead。由于amd64是CISC(complex instruction set computer)，而IR是RISC(reduced instruction set computer)，一般一条amd64的指令需要转换成多条IR指令，拿angr为例，如果将amd64指令转为VEX IR，则平均增加的指令数是4.69倍。其次，采用IR导致basic block level taint。因为由于效率的原因，从native instructions到IR的转换一般是以basic block为单位的，这样就导致无法将单个的native instruction转换成IR，所以也就只能做到哪些basic block需要符号化，而不是具体的某条指令需要符号化。这样做导致的后果就是如果某个basic block中只有一条指令和输入有关需要符号化，则整个basic block都需要符号模拟，这样就会造成很高的overhead。如果没有IR的话就可以做到指令级别的taint，就能够清楚的判断哪些指令需要符号模拟，哪些指令只需native execution，减少了不必要的符号模拟。实验表明，在一个basic block中，只有30%的指令需要符号模拟。
Ineffective Snapshot
snapshot是conclic execution常用的一个技术，它能够保存某条分支前的状态S，当该分支执行完或者”stuck”时，可以从该状态S直接执行另外一个分支，避免了重新执行的overhead。然而snapshot本身就有一些缺点：snapshot需要保存一些外部的状态(文件系统，内存管理系统)，则此时需要对影响外部状态的系统调用进行处理，一般有两个方法: full system concolic execution and External environment modeling。这两个方法都有一些缺陷：第一个方法是由于外部环境比较复杂，实现起来比较难，overhead较高；第二个则是model的system call较少，并且有些system call建模的不够完全。另外由于fuzzing的输入一般不会共享同一个分支，所以snapshot可能对于fuzzing这个场景也不是很好，所以该paper就没有采用snapshot的机制，对于每个输入都会重新执行，对于系统调用，则具体执行。
FairFuzz focus on branch coverage, it works in two main steps.
First, it identifies the program branches that are rarely hit by previously-generated inputs. It call such branches rare branches. These rare branches guard under-explored functionalities of the program. By generating more random inputs hitting these rare branches, FairFuzz greatly increases the coverage of the parts of the code guarded by them.
Second, FairFuzz uses a novel lightweight mutation technique to increase the probability of hitting these rare branches. The mutation stategy is based on the observation that certain parts of an input already hitting a rare branch are crucial to satify the conditions necessary to hit that branch. Therefore, to generate more inputs hitting the rare branch via mutation, the parts of the input that are crucial for hitting the branch should not be mutated.
Full-speed Fuzzing: Reducing Fuzzing Overhead through Coverage-guided Tracing(oakland 19)
repeat s = CHOOSENEXT(S) p = ASSIGNENERGY(s) //This paper focus for i from 1 to p do s' = MUTATE_INPUT(s) if t' crashes then add s' to Sx elseif ISINTERESTING(s') then add s' to S end if end for until timeout reached orabort-signal
P1. The DGF should define a robust distance-based mechanism that can guide the directed fuzzing by avoiding the bias to some traces and considering all traces to the targets.
P2. The DGF should strike a balance between overheads and utilities in static analysis.
P3. The DGF should select and schedule the seeds to rapidly reach target sites. AFL determines how many new inputs(i.e., “energy”) should be generated from a seed input to improve the fuzzing effectiveness(i.e., increase the coverage); this is termed “power scheduling”.
P4. The DGF should adopt an adaptive mutation strategy when the seeds cover the different program states. The desired design is that when a seed has already reached the target sites(including target lines, basic blocks or functions), it should be given less chances for coarse-grained mutations(e.g., chunk replacement).
AFLGo’s Solution
针对P1，AFLGo只是选择路径最短的那条，然而路径最短的那条可能无法触发某个漏洞。
For P2. AFLGo only considers the explicit call graph information. As a result, function pointers are treated as the external nodes which are ignored during distance calculation. Besides, AFLGo counts the same callee in tis callers only once, and it does not differentiate multiple call patterns between the caller and callee.
For P3. AFLGo applies a simulated annealing based power scheduler: it favors those seeds closer to the targets by assigning more energy to them to be mutated; the applied cooling sechedule initially assigns smaller weight on the effecte of “distance guidance”, until it reaches the “exploitation” phrase. The issue is that there is no prioritization procedure so the newly generated seeds with smaller distance may wait for a long to be mutated.
For P4. The mutation operators of AFLGo come from AFL’s two non-deterministic strategies: 1) havoc, which does purely randomly mutations such as bit flips, bytewise replace, etc; 2) splice, which generates seeds from some random byte parts of two existing seeds. Notably, during runtime AFLGo excludes all the deterministic mutation procedures and relies purely on the power scheduling on havoc/splice strategies.
Suggestions to improve DGFs:
For P1, a more accurate distance definition is needed to retain trace diversity, avoiding the focus on short traces.
For P2, both direct and indirect calls need to be analyzed; various call patterns need to be distinguished during static distance calculation.
For P3, a moderation to the current power scheduling is required. The distance-guided seed prioritization is also needed.
For P4, the DGF needs an adaptive mutation strategy, which optimally applies the fine-grained abd ciarse-graubed nytatuibs wgeb tge dustabce between the seed to the targets is different.
Hawkeye’s Design
During fuzzing, the fuzzer selects a seed from a priority seed queue. The fuzzer applies a power scheduling against the seed with the goal of giving those seeds that are considered to be “closer” to the target sites more mutation chances, i.e, energy. Specifically, this is achieved through a power function, which is a combination of the covered function similarity and the basic block trace distance. For each newly generated test seed during mutation, after capturing its execution trace, the fuzzer will calculate the covered function similarity and the basic block trace distance based on the utilities. For each input execution trace, its basic block trace distance is calculated as the accumulated basic block level distances divided by the total number of executed basic blocks; and its covered function similarity is calculated based on the overlapping of current executed functions and the target function trace closure, as well as the function level distance.
After the energy is determined, the fuzzer adaptively allocates mutation budgets on two different categories of mutations according to mutators’ granularities on the seed(coarse-grained mutations and fine-grained mutations). Afterwards, the fuzzer evaluates the newly generated seeds to prioritize those that have more energy or that have reached the target functions.
Fuzzing Machine Learning Model
TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing(18)
Most papers failed perform multiple runs, and those that did failed to account for varying performance by using a statistical test.
Many papers measured fuzzer performance not by counting distinct bugs, but instead by counting “unique crashes” using heuristics such as AFL’s coverage measure and stack hashes.
Many papers used short timeouts, without justification.
Many papers did not carefully consider the impact of seed choices on algorithmic improvements.
Papers varied widely on their choice of target programs.
]]>
<ul>
<li><p><a href="#interesting-fuzzing">Interesting Fuzzing</a></p>
<ul>
<li><a href="#coverage-based-greybox-fuzzing-as-markov-chainccs-
os tutorialhttp://bin2415.github.io/2018/09/27/os-tutorial/2018-09-28T01:17:19.000Z2018-12-23T05:33:53.018Z之前一直想了解OS的流程，直到在逛github的时候看到os-tutorial，这是一个比较适合小白了解OS，并手动构建一个OS的项目，先在此mark一下，此项目还在持续更新中~
]]>
<p>之前一直想了解OS的流程，直到在逛github的时候看到<a href="https://github.com/cfenollosa/os-tutorial" target="_blank" rel="noopener">os-tutorial</a>，这是一个比较适合小白
llvm gold buildhttp://bin2415.github.io/2018/09/27/llvm-gold/2018-09-27T23:30:17.000Z2019-07-12T15:29:22.765ZBinutils Building
]]>
<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>fuzz解析数据的库函数的方法一般是找一个简单的二进制来测试库函数的功能，通过生成不同的输入来不断地运行该二进制程序。一般是通过fork和e
AFL Fuzzing with ASANhttp://bin2415.github.io/2018/07/31/afl-asan/2018-07-31T15:05:45.000Z2018-12-23T05:33:53.000Z前言
AFL是使用比较广泛的fuzzing工具，ASAN(AddressSanitizer)是google的一个非常高效的内存错误检测工具，其能够检查出UAF,Heap/Stack buffer overflow, Use after return, Use after scope, Initialization order bugs and Memory leaks。这两者都有基于llvm的版本，所以将这两者相结合效果也是非常好的。
Since it seems to be built with ASAN and you have a restrictive memory limit configured, this is expected; please read /usr/local/share/doc/afl/notes_for_asan.txt for help
DWORD magic; // Always 0x2 DWORD offset; // Offset into the file DWORD size; // Size of the section DWORD address; // Memory address where this section will be loaded DWORD unknown; // Probably some kind of checksum?
[1] Mendez, Diego M., Ioannis Papapanagiotou, and Baijian Yang. “Internet of things: Survey on security and privacy.” arXiv preprint arXiv:1707.01879 (2017).
]]>
<h2 id="前言"><a href="#前言" class="headerlink" title="前言"></a>前言</h2><p>随着IoT(Internet of Things)设备快速增长，IoT设备的安全也逐渐引起大家的注意。如论文[1]所述，IoT的安全问题主要
