Hey guys

Hi

The discussion is based on this thread I think

https://github.com/HyperDbg/HyperDbg/issues/165

yes

I'm double checking the commit again

So, can you provide the exact commit that causes this problem ?

As you said it's somewhere in April

yeah, so this commit is the earliest I've found so far that bugchecks or hangs as soon as I do load vmm 9ee4f34dbb1a1b3db061376d1d92d868e2775d2f

You mean before this commit you didn't have the problem but after that the problem appears, right?

the latest commit I've tested and works is 728e6ea6bb2af6e8e09d9462f059cb38ef40d687

Can you send the github link of this commit ? I don't know how to map it to exact commit link in github.

https://github.com/HyperDbg/HyperDbg/commit/9ee4f34dbb1a1b3db061376d1d92d868e2775d2f

so, this is the guilty commit?

I'll try to continue the binary search

👌👌😁

Binary search always works.

Uh, so I'm conflicted to report what commit, because there are multiple bugs I'm finding that could be related or unrelated to this.

The April 5th commit, 3fc843d7abeb1810d2d698f7b05db1805ee37f15 has 2 bugs. If I do .sym reload after setting .sympath, hyperdbg-cli crashes when executing the command.

The April 1st commit here, 3fc843d7abeb1810d2d698f7b05db1805ee37f15 , I can load symbols, I can load vmm without crashing, but when I unload remove vmm, I crash after about 15-20 seconds. Tested multiple times.

using remote desktop with the physical machine

so 3fc843d7abeb1810d2d698f7b05db1805ee37f15 is the latest I've tried so far without dying on load vmm

you mean the commit after the 3fc843d7abeb1810d2d698f7b05db1805ee37f15 caused the problem. right?

b1d7415ee39f75c0b2822e0e6cccbe4b4cd6cbd3 is the earliest commit I've tested and fails.

okay, would you pls send its github link.

https://github.com/HyperDbg/HyperDbg/commit/b1d7415ee39f75c0b2822e0e6cccbe4b4cd6cbd3

there's a lot of commits between

actually only about 5, commit 16100f51f68b4feba1730b333d72f972266fcefa
Author: Behrooz Abbassi <BehroozAbbassi@outlook.com>
Date: Mon Apr 4 14:50:13 2022 -0700

Refactor: Port all CPU (VT-x) related data structures to the "IA32-doc" library.

- Control Registers
- Debug Registers
- Paging structures

commit b1d7415ee39f75c0b2822e0e6cccbe4b4cd6cbd3
Author: Behrooz Abbassi <BehroozAbbassi@outlook.com>
Date: Mon Apr 4 13:36:33 2022 -0700

Refactor: Port all CPU (VT-x) related data structures to the "IA32-doc" library.

- This commit contains lots of changes that need to be tested, so properly it's unstable.

commit 9871c1613e86fa087091f302d5c97b611346f1db
Author: Behrooz Abbassi <BehroozAbbassi@outlook.com>
Date: Mon Apr 4 10:58:59 2022 -0700

Refactor: Add "IA32-doc" library as a replacement to all IA32 data structures.

commit 340730e29fa59b55f1b782d0409dee540d3bd0ef
Merge: e4166e36 216077e8
Author: Mohammad Sina Karvandi <SinaKarvandi@users.noreply.github.com>
Date: Sat Apr 2 20:56:57 2022 +0430

Merge pull request #147 from HyperDbg/dev

Dev

commit 216077e8e722235047ff69ae8858c875dfaae3ac
Author: SinaKarvandi <ms.karvandi@yahoo.com>
Date: Sat Apr 2 20:56:16 2022 +0430

fix memory search errors in user mode addresses for debugger mode
:

to find it on github, just replace the commit hash at the end of the url with the one you want to check

yep

I'll try to narrow it down again

From your descriptions, I concluded the guilty commit is the one marked with 'x'.

I'll see if that's true

Would you please check the commit before, i mean this commit: https://github.com/HyperDbg/HyperDbg/commit/9871c1613e86fa087091f302d5c97b611346f1db

and verify if it's working or not?

it works on load vmm

That's cool, now we know that sth in https://github.com/HyperDbg/HyperDbg/commit/b1d7415ee39f75c0b2822e0e6cccbe4b4cd6cbd3 causes the errors.

sry

but, I still bugcheck after I do unload remove vmm, but that could be a separate bug.j

https://github.com/HyperDbg/HyperDbg/commit/b1d7415ee39f75c0b2822e0e6cccbe4b4cd6cbd3

this commit i mean

is it true?

9871c1613e86fa087091f302d5c97b611346f1db works on vmm load

b1d7415ee39f75c0b2822e0e6cccbe4b4cd6cbd3 fails

That's okay, tomorrow I'll try to figure out the problem of 'load vmm' on this commit.

Now, is there any other problem? other than 'load vmm' crash.

okay yes, b1d7415ee39f75c0b2822e0e6cccbe4b4cd6cbd3 is the guilty commit for load vmm. Sorry, it was a lot of commits to go through

and I can check to see if I'm able to load symbols

yep, thanks a lot for finding it 🙏

👍

wait, I'm sorry

okay yeah, 9871c1613e86fa087091f302d5c97b611346f1db does work. I saw the image I posted above that said the driver failed to install, but I guess I needed to reboot after my last load.

that's okay. what is the problem with symbol loading?

.dsymbols load fine on that commit

Does it have problem on the latest commit ?

symbols load fine on the latest commit

when I do .sym reload

after setting sympath

that's good

any other problems?

I assume it's fixed as you said it's fine in the latest commit. isn't there any other problem with .sym ?

I'm just being very specific with the steps I did

I haven't tested all cases

If I'm aware of any other bugs I'll let you know

yeah, i know

thanks

no problem

i'll try to fix that 'load vmm' error

also, do you have a crash dump for it?

yes, one moment

or if there is not crash dump, then the result of WinDbg's !analyze -v will be helpful.

this is frustrating. I'm able to load vmm on that supposed guilty commit b1d7415ee39f75c0b2822e0e6cccbe4b4cd6cbd3

actually no, it prevented me from loading cause I didn't set the sympath

I was just able to do a load vmm succesfully on the suspicious commit....

I think it's because I warm reboot, instead of from cold..

So, we still didn't find the guilty commit.

I died on it the first attempt on that commit

on load vmm instantly

but I just retried, and I successfully loaded...

😕

I'm shutting down. waiting for a minute, and doing another run

I know what you mean

These are the dark sides of developing such thing

Also, pls shutdown and retest it on the latest commit again.

Maybe it worked.

Ughhhh, it worked again...

Don't worry, I'm sure you can find it. ✌️

.

I have no idea...the latest commit also wroks now consistently...

sorry. I didn't unload *remove* vmm between those tests. So ignore that

That's great. Maybe the reason for the crash was the way you reset (cold boot or sth like this)

no it's cause I forgot to do unload remove vmm

if I don't remove, it resuses the driver that was last registered, which was in a different folder

so it used the driver from another build

Okay.

That's sth interesting

10f029b0cfd0723dbbe5ceb394ef83586b7b942a was the last commit (April 5) that I've tested and worked on vmm load

We should have a solution for this.

All in all, we can conclude that the latest commit doesn't have any problem.
For solving that issue, we should check the driver to match with the executable.

Right?

the latest commit does have a problem...

it crashes on load vmm

I didn't crash because I forgot to do unload remove vmm between tests

it was using the last successful driver build I did from another commit

🤔

This thing is super hard to figure out

Because we couldn't deterministically reproduce the error.

Btw, I'll check the crash dump tomorrow to see if I can find anything

Pls keep testing and notify us if I find any other hints

at this point I'm too invested..

I'll let you know

I've found the exact commit, 38bef6396bc98d0e1af67fc35908c42b7b3f656e. I tested it multiple times, with the following steps:

git clone --recursive https://github.com/HyperDbg/HyperDbg.git

git reset --hard 38bef6396bc98d0e1af67fc35908c42b7b3f656e

load project hyperdbg.sln into VS 2019
Set kdserial project settings to test sign, SHA256 signing
build

run hyperdbg-cli.exe
.sympath SRV*c:\Symbols*https://msdl.microsoft.com/download/symbols
.sym reload
.connect local
unload remove vmm
load vmm

as soon as I load vmm, instant bugcheck. Windows build and HW info is the same machine as what I gave in the issue tracker on github.

Dump

Unfortunately, I couldn't reproduce the error on my Skylake machine. I try other systems to see if I can reproduce it or not.

If it's possible, please also consider to check the same build on other machines. My first guess is that there might be a problem with your device. However, I try to investigate and find the cause of error.

The error is so much hardware related 😟

Nope, the error is not reproduced even in other systems. @Rwkeith

Can you test it in another system @Rwkeith ? I think your specific computer only produces this error.

Maybe I can binary search the code changes. Change only half the code, test, and then repeat until I find what the issue is. Unless the code is all linked up in a way I can't do that.

could you make a branch from this commit with reverted changes I could test? I could try myself but you'd know better what to change

So I looked through the code changes between the 2 commits. It looks clean. This tells me it could be build related. Could you send the entire build folder from that commit to me?

and tell me what steps you do, so I will do the same.

also, I'm doing this in test sign mode. and test signing kdserial. so make sure you do the same

Okay, it's not a build issue. I went to that commit, and reverted each of the files individually to the previous commit hash (7 total). I was able to run successfully after the build. My current theory is that there's a redefinition or header include order issue going on. Will debug more tomorrow.

I found the bug

CurrentPhysMask.Flags = __readmsr(IA32_MTRR_PHYSBASE0 + (CurrentRegister * 2));

Should be

CurrentPhysMask.Flags = __readmsr(MSR_IA32_MTRR_PHYSMASK0 + (CurrentRegister * 2));

I mean it should be:
CurrentPhysMask.Flags = __readmsr(IA32_MTRR_PHYSMASK0 + (CurrentRegister * 2));

with the new constants from IA32

guess you were just getting lucky on reading the msr of these other chips

Please allow me to make the commit for the work I put into it. I think I need permissions to push though?

Ah, I got it now. I did a pull request for dev

Hi, thanks a lot for sending pull request. It's merged.

Yep, we were so lucky that you find this flaw. These things are really hard to figure out.

I merged your pull request into the main (master) branch. Please check and retest the latest commit and confirm if the problem is solved or not. Thanks. 🙏

cool and thanks for the merge

🤝

I was having stability issues also when logging all syscalls back from a late February build

I haven't tested to see if it's been present in the latest build yet though.

please test it again and tell us if the problem remained unsolved.

we're currently focusing on fixing bugs related to the script engine.

sure, I plan to check it out again today.

I know we are different time zones, but that's okay

👌👌👌

there's a buffer limit on how much you can log right? I wasn't sure if that was related to my issue. I'm logging all syscalls for 1 particular process.

yes

you can change the following buffer limitations from the code :

/**
* @brief Default buffer count of packets for message tracing
* @details number of packets storage for regualr buffers
*/
#define MaximumPacketsCapacity 1000

and

/**
* @brief Size of each packet
* @details NOTE : REMEMBER TO CHANGE IT IN USER-MODE APP TOO
* @warning we redefine it on ScriptEngineEval.h change it on
* that file too
*/
#define PacketChunkSize 4096 // PAGE_SIZE

okay, and this is what I'm doing in particular. I linked you before https://pastebin.com/kgtE29Pv

okay, and did you successfully reached to your goals by the above script?

yes it logged, but my system crashed near the end after I closed the program

closed the thing I was logging. I think it was related to the amount of logging it was trying to do

because after I closed the application, it was still catching up on the console

you didn't unload before closing the program ?

the program I mean is the application I'm logging the syscalls of

🤔

BEService.exe is the program I'm monitoring syscalls of

I want to see the syscalls it makes though all the way until it closes.

ideally

The thing is, I don't know the PID of the process in advance

so I tell it to log syscalls of the whole machine and filter based on process name for event condition

have you ever used .start command ?

I haven't

can I use that on local?

https://docs.hyperdbg.org/commands/meta-commands/.start

You could use it but I disabled it for now.

Because it has some problems

but in kHyperDbg it's fully usable.

do you have a VMware Workstation?

do you test HyperDbg on your physical system?

I use hyperdbg on a physical machine at the moment.

do you attached a debugger ? Windbg?

Test sign mode

It's really weird

what is?

take a look at it :

https://docs.hyperdbg.org/commands/extension-commands/syscall#remarks

i thought PatchGuard will destroy system on test sign mode with !syscall command

hmmm. I can't recall what my bugchecks were when I did this last with the logging. Maybe I was getting patchguard checked

but running these 2 commands disables DSE

bcdedit.exe -set loadoptions DISABLE_INTEGRITY_CHECKS
bcdedit.exe -set TESTSIGNING ON

are you sure that it disables patchguard?

I don't know.

no, but those remarks state that you can disable DSE or attach a debugger

and be safe

the remark should be changed to avoid confusion.

I'll test more. I just got Hyperdbg working yesterday

I'm pretty sure disabling DSE won't turn off the PatchGuard.

I was also using an older windows 10 build when I did my old tests

👍

here it seems implied also that doing test sign disables PG

I'll add some more description for clarification.

Not getting any log data from !epthook or !syscall in the console after I paste in my hooks. I check my events list and ensure they're enabled. I do events e all. Do I need to enable console logging now? I tested my hooks on an older build and it logs immediately to console after I paste my hooks into hyperdbg-cli.

Hey, it should be fixed by now.

thanks to @Mohammadhosein_76 for fixing it

the logging for the syscalls work again on the latest commit, but !epthook 's are not running even though I've enabled them.

📨

Are you sure? How do you test it?

i retest it and it seems to be working.

okay, it's working. Maybe I didn't unload/remove the previous build properly. Thought I did it clean. Will be testing it further later

👍

👍

@Rwkeith Are you sure that the driver of HyperDbg (hprdbghv.sys) needs to be removed "unload remove vmm" each time before loading a new version of the driver? If it's not currently loaded.

I was investigating the problem you told me about. However, after getting a new update of HyperDbg and loading the driver "load vmm" the new driver is loaded (the expected behavior), so I'm wondering what the problem you told me about loading the previous version of the driver was.

I was going through several versions of hyperdbg. So this could have been the case for an older version? The way I thought it worked is that the load vmm checks to see if driver is installed, ‘sc create…’ , and whatever path was last given, was using that. I haven’t looked yet to verify, but I had some cases where I think it was still using the install path from a previous repo.

No, we didn't have any recent code updates for it.
The way you think is probably wrong. 🤔
However, it's really easy to test it. Just add a LogInfo("Test"); somewhere in the driver and test if you see the Test message or not.

alright I'm trying to figure out what I'm doing wrong here. I setup the ept hook:

!epthook nt!NtQuerySystemInformation code {5148B9A016818204F8FFFFFFD159C3}

5148B9A016688802F8FFFFFFD159C3 is

0: 51 push rcx
1: 48 b9 a0 16 68 88 02 movabs rcx,0xfffff802886816a0
8: f8 ff ff
b: ff d1 call rcx
d: 59 pop rcx
e: c3 ret

and 0xfffff802886816a0 points to my C hook in another loaded driver I call

the dump (unfortunately symbols don't exist yet I believe for this win ver)

If my assembly is:

mov rcx, rcx;
ret

no crash occurs on hook trigger

the function in C is disassembled as 2 instructions

I fixed it....changed it to

movabs rcx, 0xFFFFF802C91016A0
jmp rcx

Okay pls send a pull request and fix it in the code too.

This was just an epthook I made. This isn't to do with Hyperdbg code itself

I was troubleshooting my method on calling a C hook I created in my own code with an epthook

apologies for using this channel if it was only intended for hyperdbg development discussion

I mean, I think that what I was doing was wrong?

Oh sorry, I misunderstood

No, that's fine.

I didn't get the point. Is it fixed now? Or still has problem?

The reason why it work for you is because it's never called

you should put your hook here in newer Windowses

it does work

this shellcode didn't work for calling it though...

push rcx
movabs rcx, 0xFFFFF802C91016A0
call rcx
pop rcx
ret

what is the exact commad that you used ?

yet, this does

movabs rcx, 0xFFFFF802C91016A0
jmp rcx

!epthook nt!NtQuerySystemInformation code {48B9A01610C902F8FFFFFFE1}

I'll test it but you probably broke something in fastcall calling convention

Also for debugging it, you might use sth like :

push rcx
movabs rcx, 0xFFFFF802C91016A0
int3
call rcx
pop rcx
ret

yeah thanks for the tip

I haven't debugged it, but I imagine the additional push at the beginning may mess up the stack maybe

I just thought push/pop and call/ret would reverse each other

No, the only problem might be because of fastcall calling convention, stack is not involved because we just call the function directly from vmx root. No stack switching is involved

I'll debug it tomorrow and notify you about the results

I checked it and it was correct. Did you used sth like :
!epthook nt!NtQuerySystemInformation code {51 48 b9 a0 16 10 c9 02 f8 ff ff ff d1 59 c3 }

?

Would you please check the following assembly code :
0: 51 push rcx
1: 48 b9 a0 16 10 c9 02 movabs rcx,0xfffff802c91016a0
8: f8 ff ff
b: 48 83 ec 20 sub rsp,0x20
f: ff d1 call rcx
11: 48 83 c4 20 add rsp,0x20
15: 59 pop rcx
16: c3 ret

It's because in fastcall calling convention there is a space (i dunno the exact name, maybe it was shadow stack or sth like this) which should be freed for the callee. Maybe that's the reason you get BSOD.

I'll check this once I have a chance. I had suspicion on this shadow stack also

hey, that worked :)

thank you

👍