C# – Screen capture with Direct3D 9 API Hooks
2 votes, 5.00 avg. rating (99% score)

Since investigating screen capture techniques for Direct3D applications a year ago I have wanted to look into hooking the Direct3D API to utilise the much faster GetBackBuffer for my screen captures. Well here it is at last – a mostly managed C# solution providing easy and safe hooking of the Direct3D 9 API thanks to EasyHook, supporting both 32-bit and 64-bit.

Update: 14th March 2011 – here is a new version that supports Direct3D 10 and 11: http://spazzarama.wordpress.com/2011/03/14/c-screen-capture-and-overlays-for-direct3d-9-10-and-11-using-api-hooks/

Update: 16th April 2010 - updated code to handle fullscreen applications in a more stable manner (issue was around user inputs and Window’s getting confused about which application/window should be in the foreground). Also fixed bug where the width + height of the region to capture were needlessly being adjusted for the x1,y1,x2,y2 format of RECT – SlimDX now marshal’s the Rectangle to a RECT with this conversion for us.

Skip to download.

Overview

I have tried two approaches to hooking the Direct3D Device: “Hooking and Interface Wrapping” and “Hooking VTable function addresses”. My final implementation uses the later as it is the simpler and more effective approach for my needs.

Both approaches use “DLL injection” to directly access the address space of the target application:

DLL injection is a technique used to run code within the address space of another process by forcing it to load a dynamic-link library. (http://en.wikipedia.org/wiki/DLL_injection)

Both approaches also use function hooking, or API hooks (often referred to as a detour after the Microsoft Detours library http://research.microsoft.com/en-us/projects/detours/):

Function hooking is implemented by changing the very first few code instructions of the target function to jump to an injected [piece of] code. (http://en.wikipedia.org/wiki/Hooking)

Hooking and Interface Wrapping

This approach involves first hooking the “Direct3DCreate9” method within d3d9.dll so that we can return our own custom IDirect3D9 instance wrapping the original IDirect3D9 instance within an “Interface Wrapper” class. This allows us to intercept the call to IDirect3D9.CreateDevice that creates the IDirect3DDevice9 and in turn wrap the IDirect3DDevice9 within another “Interface Wrapper” thus providing interception for all the interface members.

An Interface Wrapper works like so:

The wrapper class implements an interface and takes an object implementing this interface as a constructor parameter. Each implemented method of the wrapper then calls the corresponding method of this original object allowing additional logic to be included before and after the original object is called.

This allows you to easily intercept and respond to each of the methods exposed by the interface. However there is a lot of extra code that needs to be created as you have to fully duplicate the original interface and in every method call the original wrapped object.

To work effectively with Direct3D this approach also requires that the host application be running and hook the “Direct3DCreate9” within the target application before the target application has finished initialising its usage of Direct3D (Direct3DCreate9 is called right at the start of any Direct3D9 application, and usually only once). This introduces potential timing issues, and does not allow you to hook a target application that is already running (a minor inconvenience really).

I stumbled across this approach to “hooking” the Direct3D API thanks to the “Direct3D StarterKit v3.0” at forums over at http://www.gamedeception.net/. There is plenty of other interesting information for dealing with Direct3D there.

Hooking VTable function addresses

This approach involves using the Virtual Method Table (VTable) of an IDirect3DDevice9 instance to determine the address of each of its virtual methods so that we are then able to install a detour around any of those that we wish to hook.

VTable is a mechanism used in a programming language to support dynamic dispatch (or run-time method binding). This supports the inheritance hierarchy whereby a virtual function points to different implementations for each sub-class (http://en.wikipedia.org/wiki/Virtual_method_table).

To access the VTable requires that we first create a temporary IDirect3DDevice9 instance within the injected code.  It is then simply a matter of obtaining a pointer to the VTable and iterating through to retrieve the method pointers.

To retrieve each of the function addresses requires that you know the order of the functions in the class (see d3d9.h in the DirectX SDK) and their parameters. Because they are functions of an object, each has an additional parameter representing “this” that you need to take into consideration when hooking.

The pros for this method are that the target application does not have to be started after the host application; the hooks can be installed at any time – and more importantly only a very small amount of native C/C++ code is required (to do the VTable address lookups), allowing the majority to be implemented in C# (It’s probably possible to do the vtable lookup directly in C# but I haven’t tried).

A con of this approach is that you have to create a temporary Direct3D device to be able to get a pointer to the VTable, which may or may not impact upon the target application (I have yet to experience any ill effects).

The provided solution here was implemented using this method after having already completed an example using the first approach. When preparing the code for this post I decided it was far too complicated and pursued the VTable route.

Stuff you need first

Some things you’ll need:

  1. Microsoft Visual Studio 2008: C++ and C# environment (install the optional 64-bit C++ compiler if required)
  2. Microsoft DirectX SDK (February 2010): headers, type libraries, examples for all DirectX versions. This is required to build the C++ helper DLL.
    http://msdn.microsoft.com/en-us/directx/aa937788.aspx
  3. EasyHook (2.6 Stable): EasyHook is an open source library released under the LGPL that supports extending (hooking) unmanaged code (APIs) with pure managed ones, from within a fully managed environment like C# using Windows 2000 SP4 and later (including 64-bit XP/Vista/Win7/Server2008).
    http://easyhook.codeplex.com/
  4. SlimDX (February 2010): SlimDX is an open source managed framework for the DirectX API supporting all manner of DirectX both 32 and 64-bit. See http://slimdx.org/features.php for the complete list. Download the “End User Runtime” or if you want the samples grab the Developer SDK: http://www.slimdx.org/download.php
  5. A Brain: needless to say all of the above is fairly useless if you don’t bring one of these along for the ride. It might also be useful if your brain has a working knowledge of C#, understands some basics of Direct3D programming and at least knows what an API hook is and why you might want to use one in the context of Direct3D applications.

Implementation

The first thing we need to do is implement the MarshalByRefObject class (referred to as the client/server interface object or simply interface) to be used by the host application and client/injected assembly to communicate via an IPC channel. We also need to create the client assembly that will be injected into the target application, all with the help of EasyHook.

For my example I want the injected assembly to take a copy of the back buffer during EndScene when requested to do so by the host application and convert it to a Bitmap object.

The marshalled object will do three things:

  1. Send a request for a screenshot from the host process to the injected assembly.
  2. Return the screenshot result from the injected assembly back to the host process.
  3. Provide a method by which the injected assembly can ping the host process and know that it needs to continue checking for screenshot requests or not.
  4. Allow debug messages from the injected assembly back to the host process.

The injection assembly will be the client for the IPC server. Using EasyHook we will inject this into the target application and call its “Run” method. This is where hooks are initialised, and a loop entered until the host application closes the IPC channel or the target process closes.

Getting the Function Addresses

The main difficulty in hooking DirectX is that the methods are not exposed in any way that allows us to get the addresses in a straight forward manner. Instead you either need to determine the addresses and hard-code them (they differ from platform to platform and version to version however), or use the VTable to lookup the addresses dynamically at runtime (which is what we will do here).

[BEGIN: AMATEURISH C++ CODE]

Here is my C++ code to retrieve the 119 addresses of IDirect3DDevice9 from the VTable and place them in an array for later retrieval:

// This will be an array of (uintptr_t*)
uintptr_t** g_deviceFunctionAddresses;
// There are 119 functions defined in the IDirect3DDevice9 interface
// (including our 3 IUnknown methods QueryInterface, AddRef, and Release)
const int interfaceMethodCount = 119;

// ...The Windows and Direct3D code to get a pointer to
// the D3D Device goes here...

// retrieve a pointer to the VTable
uintptr_t* pInterfaceVTable = (uintptr_t*)*(uintptr_t*)pd3dDevice;
g_deviceFunctionAddresses = new uintptr_t*[interfaceMethodCount];

// Retrieve the addresses of each of the methods
// (note first 3 are IUnknown methods)
// See d3d9.h IDirect3DDevice9 to see the list of methods,
// the order they appear there is the order they appear in
// the VTable, 1st one is index 0 and so on.
for (int i=0; i<interfaceMethodCount; i++) {
	g_deviceFunctionAddresses[i] = (uintptr_t*)pInterfaceVTable[i];
}

To later retrieve the address for hooking is simply a matter of returning the appropriate methods’ index in the VTable:

uintptr_t* APIENTRY GetD3D9DeviceFunctionAddress(short index)
{
	if (g_deviceFunctionAddresses) {
		return g_deviceFunctionAddresses[index];
	} else {
		return 0;
	}
}

[END: AMATEURISH C++ CODE]

If you wish to hardcode the offsets then you can determine the offsets to use as follows:

	// Determine the offset for the current platform if hardcoding addresses
	g_deviceFunctionAddresses[index] - GetModuleHandle("d3d9.dll");

You can then simply add that offset against GetModuleHandle(“d3d9.dll”) in your application and bypass the need for a C++ helper dll. The D3DHelper.dll in the attached solution logs these offsets to a log file.

The EndScene Hook

Ok now that the unmanaged code is out of the way, we are able to hook the EndScene function using EasyHook like so:

// 42 – IDirect3DDevice9.EndScene
LocalHook Direct3DDevice_EndSceneHook = LocalHook.Create(
    GetD3D9DeviceFunctionAddress(42), // This is the C++ helper method we implemented above - hardcoded addresses could be used instead
    new Direct3D9Device_EndSceneDelegate(EndSceneHook),
    this);

Within EndSceneHook we are now able to copy the BackBuffer into our own IDirect3DSurface9 with the following:

/// <summary>
/// Hook for IDirect3DDevice9.EndScene
/// </summary>
/// <param name="devicePtr">Pointer to the IDirect3DDevice9 instance. Note: object member functions always pass "this" as the first parameter.</param>
/// <returns>The HRESULT of the original EndScene</returns>
/// <remarks>Remember that this is called many times a second by the Direct3D application - be mindful of memory and performance!</remarks>
int EndSceneHook(IntPtr devicePtr)
{
    using (Device device = Device.FromPointer(devicePtr))
    {
        // If you need to capture at a particular frame rate, add logic here decide whether or not to skip the frame
        try
        {
            // Is there a screenshot request? If so lets grab the backbuffer
            lock (_lockRenderTarget)
            {
                if (_screenshotRequest != null)
                {
                    DateTime start = DateTime.Now;
                    try
                    {
                        // First ensure we have a Surface to the render target data into
                        if (_renderTarget == null)
                        {
                            // Create offscreen surface to use as copy of render target data
                            using (SwapChain sc = device.GetSwapChain(0))
                            {
                                _renderTarget = Surface.CreateOffscreenPlain(device, sc.PresentParameters.BackBufferWidth, sc.PresentParameters.BackBufferHeight, sc.PresentParameters.BackBufferFormat, Pool.SystemMemory);
                            }
                        }

                        using (Surface backBuffer = device.GetBackBuffer(0, 0))
                        {
                            // Create a super fast copy of the back buffer on our Surface
                            device.GetRenderTargetData(backBuffer, _renderTarget);

                            // We have the back buffer data and can now work on copying it to a bitmap

                            // NOTE: originally I had tried calling ProcessRequest in a separate
                            // thread, however I ran into stability issues resulting in
                            // corrupt images or memory violation issues. Therefore ProcessRequest
                            // is called direct.
                            // ProcessRequest is also the slowest part of the EndScene hook.
                            ProcessRequest();
                        }
                    }
                    finally
                    {
                        // We have completed the request - mark it as null so we do not continue to try to capture the same request
                        // Note: If you are after high frame rates, consider implementing buffers here to capture more frequently
                        //         and send back to the host application as needed. The IPC overhead significantly slows down
                        //         the whole process if sending frame by frame.
                        _screenshotRequest = null;
                    }
                    DateTime end = DateTime.Now;
                    _lastScreenshotTime = (end - start);
                }
            }
        }
        catch
        {
            // If there is an error we do not want to crash the hooked application, so swallow the exception
            // TODO: log any exceptions
        }

        // EasyHook has already repatched the original EndScene - so calling EndScene against the SlimDX device will call the original
        // EndScene and bypass this hook. EasyHook will automatically reinstall the hook after this hook function exits.
        return device.EndScene().Code;
    }
}

The ProcessRequest method copies the data from our copy of the back buffer into a stream that we can now transfer back to the host process within a separate thread:

/// <summary>
/// Copies the _renderTarget surface into a stream and starts a new thread to send the data back to the host process
/// </summary>
void ProcessRequest()
{
    if (_screenshotRequest != null)
    {
        // We need to convert Width and Height to x2, and y2 respectively, because
        // SlimDX Surface.ToStream(...) expects the rectangle to have x1,y1,x2,y2 not x1,y1,width,height
        Rectangle region = new Rectangle(_screenshotRequest.RegionToCapture.Location, _screenshotRequest.RegionToCapture.Size);
        region.Width += region.X;
        region.Height += region.Y;

        // Prepare the parameters for RetrieveImageData to be called in a separate thread.
        RetrieveImageDataParams retrieveParams = new RetrieveImageDataParams();

        // After the Stream is created we are now finished with _renderTarget and have our own separate copy of the data,
        // therefore it will now be safe to begin a new thread to complete processing.
        // Note: RetrieveImageData will take care of closing the stream.
        // Note 2: Surface.ToStream is the slowest part of the screen capture process - the other methods
        //         available to us at this point are _renderTarget.GetDC(), and _renderTarget.LockRectangle/UnlockRectangle
        if (_screenshotRequest.RegionToCapture.Width == 0)
        {
            // The width is 0 so lets grab the entire window
            retrieveParams.Stream = Surface.ToStream(_renderTarget, ImageFileFormat.Bmp);
        }
        else if (_screenshotRequest.RegionToCapture.Height > 0)
        {
            retrieveParams.Stream = Surface.ToStream(_renderTarget, ImageFileFormat.Bmp, region);
        }

        if (retrieveParams.Stream != null)
        {
            // _screenshotRequest will most probably be null by the time RetrieveImageData is executed
            // in a new thread, therefore we must provide the RequestId separately.
            retrieveParams.RequestId = _screenshotRequest.RequestId;

            // Begin a new thread to process the image data and send the request result back to the host application
            Thread t = new Thread(new ParameterizedThreadStart(RetrieveImageData));
            t.Start(retrieveParams);
        }
    }
}

To ensure that _renderTarget is always in the correct format we also have to hook IDirect3DDevice9.Reset so that we can dispose of _renderTarget and ensure it is recreated with the correct parameters (e.g. to handle switching between windowed and full screen).

/// <summary>
/// Reset the _renderTarget so that we are sure it will have the correct presentation parameters (required to support working across changes to windowed/fullscreen or resolution changes)
/// </summary>
/// <param name="devicePtr"></param>
/// <param name="presentParameters"></param>
/// <returns></returns>
int ResetHook(IntPtr devicePtr, ref D3DPRESENT_PARAMETERS presentParameters)
{
    using (Device device = Device.FromPointer(devicePtr))
    {
        PresentParameters pp = new PresentParameters()
        {
            AutoDepthStencilFormat = (Format)presentParameters.AutoDepthStencilFormat,
            BackBufferCount = presentParameters.BackBufferCount,
            BackBufferFormat = (Format)presentParameters.BackBufferFormat,
            BackBufferHeight = presentParameters.BackBufferHeight,
            BackBufferWidth = presentParameters.BackBufferWidth,
            DeviceWindowHandle = presentParameters.DeviceWindowHandle,
            EnableAutoDepthStencil = presentParameters.EnableAutoDepthStencil,
            FullScreenRefreshRateInHertz = presentParameters.FullScreen_RefreshRateInHz,
            Multisample = (MultisampleType)presentParameters.MultiSampleType,
            MultisampleQuality = presentParameters.MultiSampleQuality,
            PresentationInterval = (PresentInterval)presentParameters.PresentationInterval,
            PresentFlags = (PresentFlags)presentParameters.Flags,
            SwapEffect = (SwapEffect)presentParameters.SwapEffect,
            Windowed = presentParameters.Windowed
        };
        lock (_lockRenderTarget)
        {
            if (_renderTarget != null)
            {
                _renderTarget.Dispose();
                _renderTarget = null;
            }
        }
        // EasyHook has already repatched the original Reset so calling it here will not cause an endless recursion to this function
        return device.Reset(pp).Code;
    }
}

Running It

The following assemblies must be registered within the GAC before hooking will work:

  1. SlimDX.dll
  2. EasyHook.dll
  3. The assembly containing your IPC object (i.e. ScreenshotInterface in ScreenshotInject.dll)
  4. Also put the assembly that will be injected to the target application, in my case the same as the IPC assembly (i.e.ScreenshotInject.dll)

OR: EasyHook has a handy method for doing this at runtime if your host application is running as an Administrator, e.g.:

// Note: EasyHook.dll will be automatically added to GAC if not already when using the following method
Config.Register(
  "My Injection assemblies",
  "Assembly1.dll",
  "Assembly2.dll");

You need to have the EasyHook binaries within the same directory as your applications executable (see the attached solutions bin directory).

I prefer to use the following post build command to register my injection assembly in the GAC, at least while debugging. It requires that you are running Visual Studio as an Administrator:

"C:Program FilesMicrosoft SDKsWindowsv6.0ABingacutil.exe" /i "$(TargetPath)"

You should already have the Direct3D SDK. You may need to adjust the locations in the D3DHelper DLL project (I installed it to “C:DirectX SDK“). The examples included in the SDK are handy for testing (e.g. checking that you have not tied up any Direct3D resource after everything closes down and so on). If you are on a 64-bit OS, the SDK includes 64-bit examples so you can test if you can’t find another 64-bit Direct3D application. You will notice that if you exit the target application first there is a Surface object that Direct3D samples complain about not having been freed correctly – if you close the host application first you will not get this warning.

64-bit Support

The .NET assemblies all support 64-bit as they are compiled for “Any CPU”. The only difference is that you need to compile the helper C++ dll in 64-bit for it to work when hooking 64-bit applications (other than the Direct3D SDK example I have not come across any yet).

Improvements, Considerations and What Next?

If you are looking to capture frames for a movie, do the buffering in the injected assembly, I believe you will be able to achieve fairly high frame rates. Then return the buffer to the host process as needed.

For completeness you may want to investigate an alternative to SlimDX Surface.ToStream – this utilises the underlying D3DXSaveSurfaceToFileInMemory. Instead you could try Surface.LockRectangle and read the pixels directly. I have yet to try this and compare performance.

Important!: Remember that your injected assembly is running in the target application. Handle ALL your exceptions and cleanup after yourself. Be a good citizen and don’t cause any unnecessary interruption to the target process. Direct3D hooking involves functions that are called many many times per second, so keep this in mind at all times!

Note that I pass around a byte[] for my IPC not a Bitmap object, this is because the Bitmap class does not support MarshalByRefObject.

If you want to implement in-game overlays, simply prepare everything within EndScene, and add it to the scene as necessary before passing on the request to the “real” EndScene method.

What about Direct3D 10/11? I believe a similar process will work fine, and will be preparing an updated example to support all current versions in the future.

Further Reading

I recommend taking a look through the EasyHook documentation and examples to get a better understanding of how it hangs together.

The SlimDX documentation and examples will be handy if you want to implement any additional features such as in-game overlays.

Greg over at Ring3 Circus has some great articles that have helped with my understanding of DLL injection and Direct3D hooking: http://www.ring3circus.com/category/gameprogramming/

The forums over at gamedeception.net have been a great learning resource. You will have to register to view the posts and download the example source code (all C++): http://forum.gamedeception.net/forum.php

As always searching on Google found plenty of useful information too numerous to remember or list here.

Feedback

I have spent some time trying to get this right so please leave a comment if you can think of any improvements or have any other feedback.

I am always interested in hearing about how this has been useful in your own solutions. You can use the contact page or leave a comment here if you would like to share.

Download

C# Direct3D 9 Hook source download (revision 2 - 2010-04-16)

The original version of the source code can still be found here.

C# – Screen capture with Direct3D 9 API Hooks
2 votes, 5.00 avg. rating (99% score)